Topic: Store Lots of Data in Text File

[mailmaverick]

Hi All

I want to store lots of data in form of List of Lists in a Text File and I want to store it in such a way that I am able to retrieve it fastest (since there is lots of data).

My data is in following form of multiple lists, each list containing further lists having three coordinates (coordinates will always be integers only). Number of sub-lists in a main-list may vary and also number of main-lists is also dynamic :-

'( '(45 63 23) '(74 2 52) '(45 34 64)...........)           ; This is 1 main-list containing many sub-lists. Each sub-list contains 3 coordinates.
'( '(85 96 23) '(93 85 245) '(864 235 673)...........)
'( '(75 23 95) '(23 66 33) '(23 96 90)...........)
'( '(58 18 19) '(27 93 60) '(61 80 504)...........)
------
------

Kindly help me with fastest store and retrieve functions. Thanks a lot.

[Kerry]

See what the data size limit is for something like these.
The code is pretty self explanatory.

Code - Auto/Visual Lisp: [Select]

1.  
2. ;;;--------------------------------------------------------------------
3. ;;;------------------------------------------------------------------
4. ;;;
5.  
6. ;; edit to suit .....
7. ;; (KDUB:TextFile-to-List "F:\\project\\test0625\\DWG_LIST.PRN")
8.  
9. (defun kdub:textfile-to-list
10.        (filename / fileobject filefound filesystem openfileas returnvalue)
11.   (if (setq filefound (findfile filename))
12.     (progn (setq filesystem (vlax-create-object "Scripting.FileSystemObject")
13.                  fileobject (vlax-invoke filesystem "GetFile" filefound)
14.                  openfileas (vlax-invoke fileobject "OpenAsTextStream" 1 0)
15.            )
16.            (while (= (vlax-get openfileas "AtEndOfStream") 0)
17.              (setq returnvalue
18.                     (cons (vlax-invoke openfileas "ReadLine") returnvalue)
19.              )
20.            )
21.            (vlax-invoke openfileas "Close")
22.            (vlax-release-object openfileas)
23.            (vlax-release-object fileobject)
24.            (vlax-release-object filesystem)
25.            (reverse returnvalue)
26.     )
27.     nil
28.   )
29. )
30.  
31.  
32. (defun kdub:list-to-textfile (filename      datalist      /
33.                               fileobject    filefound     filesystem
34.                               openfileas    returnvalue
35.                              )
36.   (if (setq filefound (findfile filename))
37.     (progn (setq filesystem (vlax-create-object "Scripting.FileSystemObject")
38.                  fileobject (vlax-invoke filesystem "GetFile" filefound)
39.                  openfileas (vlax-invoke fileobject "OpenAsTextStream" 2 0)
40.            )
41.            (foreach line datalist (vlax-invoke openfileas "WriteLine" line))
42.            (vlax-invoke openfileas "Close")
43.            (vlax-release-object openfileas)
44.            (vlax-release-object fileobject)
45.            (vlax-release-object filesystem)
46.            t
47.     )
48.     nil
49.   )
50. )
51.  

Read the file into a variable ..
foreach on the variable to access each sub-list.

[ymg]

mailmaverick,

Why do you have a quote in front of each sublist ?

What would be wrong with the following?

When you say huge, how huge ?

What Kerry posted will work for sure, but is not necessary
unless you hit the limit of text file.

There were also this by MP Read and Write Stream which is very fast.

Code: [Select]

(setq lst '(((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
   )
)
(setq f (open "test.txt" "w"))   
  (princ lst f)                         
(close f)

(setq lst nil)


(setq f (open (findfile "test.txt") "r"))   
  (setq lst (read  (read-line f)))               
(close f)


[David Bethel]

I use this scenario a lot :


TEST.DAT

Code: [Select]

(setq data '(
    ((45 63 23)(74 2 52)(45 34 64))
    ((85 96 23)(93 85 245)(864 235 673))
    ((75 23 95)(23 66 33) (23 96 90))
    ((58 18 19) 27 93 60) (61 80 504))
))


Code: [Select]

(load "TEST.DAT")

-David

[mailmaverick]

mailmaverick,

Why do you have a quote in front of each sublist ?

YMG, It is my mistake. I have added it to just show the sublists.

What would be wrong with the following?

YMG, I think your code is OK.

When you say huge, how huge ?

YMG, huge can be that there can be around 10,000 lists, each containing around 10,000 sublists.

What Kerry posted will work for sure, but is not necessary
unless you hit the limit of text file.

There were also this by MP Read and Write Stream which is very fast.

I want to ask which method is faster, YMGs or Kerrys ?

[Kerry]

YMG, huge can be that there can be around 10,000 lists, each containing around 10,000 sublists.

I want to ask which method is faster, YMGs or Kerrys ?

How about you do your part  and provide sample data that is a realistic representation of the files you use ??

[Kerry]

For a file containing a list
of 40 lists
of 21 lists
of 3 lists
of 3 elements

ie 7560 integers
Size on Disk : 32.0 KB (32,768 bytes)

Benchmarking-V2 :B=500 [M.P.2005 <revised kdub 2005,2014>] ...........
Elapsed milliseconds for 256 iteration(s)/ relative Timing :

    (KDUB:TEXTFILE-TO-LIST FN).....1000 / 1.8832 <slowest>: 3.90625000ms Per iteration
    (DO1 FN)........................859 / 1.6177
    (_READSTREAM FN 0)..............531 / 1.0000 <fastest>: 2.07421875ms Per iteration

Be aware that _READSTREAM  returns a string, which will require translation

[Kerry]

Regarding _readStream :

If the translation is done by reading the string returned ; ie :

Code - Auto/Visual Lisp: [Select]

1. (read (_ReadStream fn 0))

the performance changes significantly.

Code - Auto/Visual Lisp: [Select]

1. (setq fn "test.txt")
2.  
3. (defun do1 (fn / f lst)
4.   (setq f (open (findfile fn) "r"))
5.   (setq lst (read (read-line f)))
6.   (close f)
7.   lst
8. )
9.  
10. (Benchmark2
11.   500
12.   '((do1 fn)
13.     (read (_ReadStream fn 0))
14.     (kdub:textfile-to-list fn)
15.    )
16. )

Benchmarking-V2 :B=500 [M.P.2005 <revised kdub 2005,2014>] ...........
Elapsed milliseconds for 256 iteration(s)/ relative Timing :

    (READ (_READSTREAM FN 0))......1203 / 1.4254 <slowest>: 4.69921875ms Per iteration
    (KDUB:TEXTFILE-TO-LIST FN)......907 / 1.0746
    (DO1 FN)........................844 / 1.0000 <fastest>: 3.29687500ms Per iteration

[Marc'Antonio Alessi]

I use this scenario a lot :
...
(load "TEST.DAT")
-David

Me too:

Code: [Select]

(defun ALE_UtlCfg_WriteData (DatNam LspOut / FilPnt TmpVal SysTim)
      (setq FilPnt (open LspOut "w"))
      (write-line (strcat "(setq " (vl-symbol-name DatNam) " '") FilPnt)
      (princ (vl-symbol-value DatNam) FilPnt)
      (write-line ")" FilPnt)
      (close FilPnt)
)
(setq #lst
   '(((45 63 23) (74 2 52) (45 34 64))
         ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))         
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))         
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
    ((45 63 23) (74 2 52) (45 34 64))           
            ((85 96 23) (93 85 245) (864 235 673))
            ((75 23 95) (23 66 33) (23 96 90))
            ((58 18 19) (27 93 60) (61 80 504))
   )
)


Code: [Select]

Comando: (ALE_UtlCfg_WriteData '#lst "test.lsp")
nil
Comando: (setq #lst nil)
nil
Comando: (load "test.lsp")
(((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)))
Comando: !#lst
(((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)) ((45 63 23) (74 2 52) (45 34 64)) ((85 96 23) (93 85 245) (864 235 673)) ((75 23 95) (23 66 33) (23 96 90)) ((58 18 19) (27 93 60) (61 80 504)))


[Marc'Antonio Alessi]

Thi is better if strings in the list:

Code: [Select]

(defun ALE_Utl_WriteData (DatNam LspOut / FilPnt)
      (setq FilPnt (open LspOut "w"))
      (write-line (strcat "(setq " (vl-symbol-name DatNam) " '") FilPnt)
      (prin1 (vl-symbol-value DatNam) FilPnt) ; < prin1
      (write-line ")" FilPnt)
      (close FilPnt)
)

[ymg]

YMG, huge can be that there can be around 10,000 lists, each containing around 10,000 sublists.

So we are talking, 8 to  10 megs in the file assuming that the sublist are integers.

Kerry's or MP's solution should be worth it.

Kerry's has the advantage of not requiring the extra translation step.

ymg

[mailmaverick]

Thanks to all. My problem has been solved. I have finally used YMG's solution and its working perfect !

Thanks also to Kerry for his excellent inputs.

[VovKa]

hardcoding variable name into lsp data file is not a good idea
as soon as load returns the last expression evaluated, one can do it this way

Code: [Select]

;write
(setq l '(1 2 3 "a" (4 5 (6))))
(setq f (open "c:\\temp\\test.lsp" "w"))
(princ "'" f)
(prin1 l f)
(close f)

;read
(setq l (load "c:\\temp\\test.lsp"))


[Marc'Antonio Alessi]

hardcoding variable name into lsp data file is not a good idea
as soon as load returns the last expression evaluated, one can do it this way

what is the difference to say that it is not a good idea? The example I posted is what I use to update a global variable whose value, if it changes, is written to a file lsp and reloaded into new file dwg if opened.

[Marc'Antonio Alessi]

hardcoding variable name into lsp data file is not a good idea
as soon as load returns the last expression evaluated, one can do it this way

I just wanted to show that the load is faster than the read and that is longer a nightmare in file containing a list of 10000 lists of 10000 lists (about 1 Gb) as request by OP:

"YMG, huge can be that there can be around 10,000 lists, each containing around 10,000 sublists."

Code: [Select]

(progn
(setq alist
'((45 63 23)(74 2 52)(45 34 64)(45 63 23)(74 2 52)(45 34 64)(45 63 23)(74 2 52)(45 34 64)(74 2 52))
)
(setq alist2 alist)
(setq alist2 (repeat 999 (setq alist2 (append alist2 alist))))
(princ (length alist2)) (terpri)
(setq alist (list alist2))
(setq alist (repeat xxxx (setq alist (cons alist2 alist))))
(princ (length alist)) (terpri)
(princ (length (car alist))) (terpri)
)


Code: [Select]

Benchmark.lsp | © 2005 Michael Puckett | All Rights Reserved
For a file containing a list of 100 lists of 10000 lists:
Elapsed milliseconds / relative speed for 2 iteration(s):
    (ALE_UTL_LOADDATA "z:\\temp\\Ale.lsp").....1343 / 5.36 <fastest>
    (DO1 "z:\\temp\\Ale.lsp")..................7203 / 1 <slowest>

For a file containing a list of 200 lists of 10000 lists:
Elapsed milliseconds / relative speed for 1 iteration(s):
    (ALE_UTL_LOADDATA "z:\\temp\\Ale.lsp")......1688 / 27.43 <fastest>
    (DO1 "z:\\temp\\Ale.lsp")..................46297 / 1 <slowest>
for bigger list I can load a file containing a list of 300 lists of 10000 lists (about 31 Mb) but fail with a file containing a list of 500 lists of 10000 lists (about 51 Mb). Read method seems to fail before...

[roy_043]

hardcoding variable name into lsp data file is not a good idea
as soon as load returns the last expression evaluated, one can do it this way

what is the difference to say that it is not a good idea? The example I posted is what I use to update a global variable whose value, if it changes, is written to a file lsp and reloaded into new file dwg if opened.

I agree Vovka here. Keeping the variable outside the data file is better for two reasons:
1. It is more flexible: You can attach the same data to multiple variables.
2. You are less likely to end up with unlocalized variables.

[Marc'Antonio Alessi]

...
I agree Vovka here. Keeping the variable outside the data file is better for two reasons:
1. It is more flexible: You can attach the same data to multiple variables.
2. You are less likely to end up with unlocalized variables.

I agree with you... but before you say it's a bad idea...

Code: [Select]

(defun c:test ( / alist)
  (setq alist '((45 63 23)(74 2 52)(45 34 64)))
  (ALE_Utl_WriteData 'alist "z:\\temp\\Ale.lsp")
  (ALE_Utl_LoadData "z:\\temp\\Ale.lsp")
  (princ)
)
Comando: TEST

Comando: !alist
nil
> You can localize the variable alist.

[Kerry]

...
I agree Vovka here. Keeping the variable outside the data file is better for two reasons:
1. It is more flexible: You can attach the same data to multiple variables.
2. You are less likely to end up with unlocalized variables.

I agree with you... but before you say it's a bad idea...

Code: [Select]

(defun c:test ( / alist)
  (setq alist '((45 63 23)(74 2 52)(45 34 64)))
  (ALE_Utl_WriteData 'alist "z:\\temp\\Ale.lsp")
  (ALE_Utl_LoadData "z:\\temp\\Ale.lsp")
  (princ)
)
Comando: TEST

Comando: !alist
nil
> You can localize the variable alist.

I'd do it thusly :

In the function that reads the data make the last statement return the list you've read

and assign the returned value to a variable you have control over ...

<...>
(setq myData (readTheDataFile fileName) )
<... >

[Marc'Antonio Alessi]

...
I'd do it thusly :
In the function that reads the data make the last statement return the list you've read
and assign the returned value to a variable you have control over ...
<...>
(setq myData (readTheDataFile fileName) )
<... >

Okay I agree, but I think it's more important to discuss whether it is best to read the data file or load it as lisp file, also because this suggestion seems to have been ignored...