php - How can i reduce my text file size? -

i have php code gets product ids database (23436 unique records).

i fetch each product id , check if has been set in feature_product table comparing productids.

if no records found under id in features tables check trial.txt file missing features of products again comparing productid in text file productid not present in feature_product table.

the problem trial.txt file has 593262 lines in , takes forever match productid in file. run out of memory. took me 15 hours data off file , in parts manually. there way make faster or not run out of time , memory?

i tried increasing max execution time in php.ini file suggested posts on sites. keeps running out of memory or max execution time. using mysqli once right, mysql not used more. thought of dividing product ids can loop 5000 @ time don't think in execution time.

<?php $conn = mysql_connect("localhost", "dbuser", "pwd");     //loop through 1st line avoid headers in csv  if (!$conn){   die('could not connect : ' . mysql_error());   echo mysql_error();  }    echo '<p>connected!';   mysql_select_db("mydb") or die( "unable select database");  //select product ids product table product array  $parray = mysql_query("select `id_product` `product`",$conn);   //loop through each product id  while($row = mysql_fetch_assoc($parray)) {   //get product id check if exists in features table  $productid = $row["id_product"];  //check whether product id exists in feature table product_id matches both product table , features table  $farray = mysql_query("select * `feature_product` `id_product`=$productid");  //if product id not have entry in feature table call function check if product id has features in text file if(mysql_num_rows($farray) ==0)  {  checkfeatures($productid);  }  else continue; }  function checkfeatures($productid){ //trial.txt contains features of products missing in features table products in products table $fd = fopen('trial.txt', 'r'); $fheader = fgets($fd);   //creates new text file save features(multiple records per product) separated ',' future use $my_file = 'file.txt'; $handle = fopen($my_file, 'a') or die('cannot open file:  '.$my_file);  while (($data = fgetcsv($fd,0, "~")) !== false) { //since text file has many products ones missing in features table comparing product id 1st element of data array      if($data[0]==$productid){      $d= $data[0].",".$data[1].",".$data[2].$data[3]."\n";      echo $d."<br/>";      fwrite($handle, $d);      }   } fclose($fd); fclose($handle);     } ?>   

example of product table

id_product,shop,manufacutrer,category   1000010,1,41,1112,1   1000011,1,7,1721,1   1000012,1,7,1721,1   

example of feature table

feature_id,id_product,value   1,1000010,1   3,1000010,2   6,1000011,5   11,1931555,1  

sample trial.txt

imsku~attributeid~value~unit~storedvalue~storedunit   1000006~16121~2-25~~~   1000006~3897~* mcafee protection suite~~~   1000006~3933~* 1yr subscription~~~   1000010~1708~feb 2011~~~   1000010~1710~cisco~~0.00~   1000010~1711~http://www.cisco.com~~~   1000011~2852~1~~0.00~   1000011~2855~light cyan~~0.00~   1000012~2840~may 2010~~~   1000012~2842~hp~~0.00~   

i tried load text file table in sql suggested users

<?php  $con=mysqli_connect("localhost","username","pwd","db"); // check connection if (mysqli_connect_errno()) { echo "failed: " . mysqli_connect_error(); }  mysqli_query($con,"create table if not exists `add_features` (`id_product` int(10) not null, `id_feature` int(10) not null, `value` varchar(255),`unit` varchar(20),`s_value` varchar(20),`s_unit` varchar(20))");  $sql = "load data infile 'trial.txt' table `add_features` fields terminated '~' "; if ($con->query($sql) === true) { echo "ok!"; } else { echo "error: " . $sql . "<br>" . $con->error; }  $result = mysqli_query($con,"select * `add_features`");  echo "<table class='add_features'> <tr class='titles'> <th>product_id</th> <th>feature_id</th> <th>value</th> <th>unit</th> </tr>";  while($row = mysqli_fetch_array($result)) { echo "<tr>"; echo "<td>" . $row['id_product'] . "</td>"; echo "<td>" . $row['id_feature'] . "</td>"; echo "<td>" . $row['value'] . "</td>"; echo "<td>" . $row['unit'] . "</td>"; echo "</tr>"; } echo "</table>";  mysqli_close($con); ?>   

but getting error
error: load data infile 'trial.txt' table add_features fields terminated '~'

if trial.txt file static, process / parse either separate smaller files based on logical divider, or import new database table (preferable) searching instant. it's 1 time import , it's done.

if it's not static, how change?