Author Topic: How to improve the computing speed of MD5  (Read 4333 times)

0 Members and 1 Guest are viewing this topic.

baitang36

  • Bull Frog
  • Posts: 213
How to improve the computing speed of MD5
« on: October 16, 2022, 09:28:21 PM »
The MD5 algorithm of the lsp version is very slow, which is criticized by many people. Can you optimize it and improve the speed?

Yc-MD5.lsp is the source code of MD5 algorithm for lsp version found in the forum. It has been changed from time to time with little speed improvement. It seems that we need to find another way out.

The problem with the lsp version of MD5 is that it uses lsp to build wheels and uses lsp statements to implement MD5.

After searching, it is found that the Windows kernel has MD5 algorithm, that is, it provides the API of MD5 algorithm, so long as it is called, there is no need to make your own wheel.

syz-md5.fas is implemented by calling the system api with pure lsp. 32-bit and 64 bit are common. The usage is after loading (syz-md5 "string"). The return value is the MD5 value string of the string. The speed can be described as fast. In my test, 500 MD5 calculations were performed, and the yc-md5 program took 14.15 seconds; The syz-md5 program takes 0.032 seconds, and the speed difference is 400 times.

Tt5.lsp is the test program.
Command: (LOAD "D:/aa/tt5. lsp") C: TT5
Command: tt5
This method is only applicable to autocad because it uses the unexposed functions of autocad

kdub_nz

  • Mesozoic keyThumper
  • SuperMod
  • Water Moccasin
  • Posts: 2087
  • class keyThumper<T>:ILazy<T>
Re: How to improve the computing speed of MD5
« Reply #1 on: October 17, 2022, 02:11:25 AM »
500 MD5 calculations seems a lot.

Called Kerry in my other life
Retired; but they dragged me back in !

I live at UTC + 13.00

---
some people complain about loading the dishwasher.
Sometimes the question is more important than the answer.

d2010

  • Bull Frog
  • Posts: 322
Re: How to improve the computing speed of MD5
« Reply #2 on: October 17, 2022, 11:13:07 AM »
The MD5 algorithm of the lsp version is very slow, which is criticized by many people. Can you optimize it and improve the speed?
Yc-MD5.lsp is the source code of MD5 algorithm for lsp version found in the forum. It has been changed from time to time with little speed improvement. It seems that we need to find another way out.
For optimize the code source.lisp, then you  must learn, before, other programming  language. (e.g Java, Julia, ). Why?
Because .. The module javac.exe contain hunders * hunders of optimizations.
Too many years, 10-50 engineers work at
--a)code optimization of speed
--b)code optimization of eating memory
--c)code optimization of parse the variabile (local, static ,global , dynamical, incapsulate inside Object)
Ps:=Visual Lisp are too weak for support the optimizations.
 :sleezy:
« Last Edit: October 17, 2022, 11:17:04 AM by d2010 »

VovKa

  • Water Moccasin
  • Posts: 1617
  • Ukraine
Re: How to improve the computing speed of MD5
« Reply #3 on: October 17, 2022, 01:53:50 PM »
syz-md5.fas is implemented by calling the system api with pure lsp.
and this is great news, baitang36
well done
i think it brings lisp to a new level

domenicomaria

  • Swamp Rat
  • Posts: 720
Re: How to improve the computing speed of MD5
« Reply #4 on: October 17, 2022, 02:00:06 PM »
syz-md5.fas is implemented by calling the system api with pure lsp
when this possibility will be available for the normal people ? :-)

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #5 on: October 18, 2022, 07:43:17 PM »
syz-md5.fas is implemented by calling the system api with pure lsp
when this possibility will be available for the normal people ? :-)
Because this technology is not disclosed by autodesk, I am afraid that if it is disclosed, it will be blocked and will not be used in the future

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #6 on: October 18, 2022, 07:44:58 PM »
syz-md5.fas is implemented by calling the system api with pure lsp.
and this is great news, baitang36
well done
i think it brings lisp to a new level
I tested that there was no problem in the 32-bit system. However, in 64 bit systems, some APIs will make mistakes

domenicomaria

  • Swamp Rat
  • Posts: 720
Re: How to improve the computing speed of MD5
« Reply #7 on: October 18, 2022, 10:55:19 PM »
Quote
Because this technology is not disclosed by autodesk, I am afraid that if it is disclosed, it will be blocked and will not be used in the future

but maybe with Bricscad it could be different ...

VovKa

  • Water Moccasin
  • Posts: 1617
  • Ukraine
Re: How to improve the computing speed of MD5
« Reply #8 on: October 19, 2022, 05:26:15 AM »
I tested that there was no problem in the 32-bit system. However, in 64 bit systems, some APIs will make mistakes
i guess the api function that generates MD5 hash works with strings, takes a string as an argument and returns a string
what about arrays?
is there a way to pass an array as an argument to dll calls?

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #9 on: October 19, 2022, 07:19:27 AM »
I tested that there was no problem in the 32-bit system. However, in 64 bit systems, some APIs will make mistakes
i guess the api function that generates MD5 hash works with strings, takes a string as an argument and returns a string
what about arrays?
is there a way to pass an array as an argument to dll calls?
In fact, the input of md5 is a pile of binary data, and the output is also binary. It is made into a string for reading convenience
The array can be converted to a string first, and then MD5 can be calculated

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #10 on: October 19, 2022, 07:24:54 AM »
syz-md5.fas is implemented by calling the system api with pure lsp.
and this is great news, baitang36
well done
i think it brings lisp to a new level
  Called ntdll.dll, core code is :
(syz-api "ntdll.dll" "MD5Init" ctx)
  (syz-api "ntdll.dll" "MD5Update" ctx str1  (strlen str1)  )
  (syz-api "ntdll.dll" "MD5Final" ctx)

VovKa

  • Water Moccasin
  • Posts: 1617
  • Ukraine
Re: How to improve the computing speed of MD5
« Reply #11 on: October 19, 2022, 08:10:55 AM »
In fact, the input of md5 is a pile of binary data, and the output is also binary. It is made into a string for reading convenience
The array can be converted to a string first, and then MD5 can be calculated
this is clear
what if i want to pass to a dll function an array of floats (list of point coordinates for example)?
is it possible with syz-api?

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #12 on: October 19, 2022, 08:34:40 AM »
In fact, the input of md5 is a pile of binary data, and the output is also binary. It is made into a string for reading convenience
The array can be converted to a string first, and then MD5 can be calculated
this is clear
what if i want to pass to a dll function an array of floats (list of point coordinates for example)?
is it possible with syz-api?
(syz-md5(strcat(rtos x)(rtos y)(rtos z)))

VovKa

  • Water Moccasin
  • Posts: 1617
  • Ukraine
Re: How to improve the computing speed of MD5
« Reply #13 on: October 19, 2022, 09:09:03 AM »
(syz-md5(strcat(rtos x)(rtos y)(rtos z)))
a have a library of functions written i C++ and compiled to a dll file (i can not modify it)
i want to call one of these functions from lisp and pass two arguments to it, one is a float and another is an array of floats
can it be done with your syz-api function?

baitang36

  • Bull Frog
  • Posts: 213
Re: How to improve the computing speed of MD5
« Reply #14 on: October 19, 2022, 09:50:09 PM »
(syz-md5(strcat(rtos x)(rtos y)(rtos z)))
a have a library of functions written i C++ and compiled to a dll file (i can not modify it)
i want to call one of these functions from lisp and pass two arguments to it, one is a float and another is an array of floats
can it be done with your syz-api function?
syz-api only support string and inter(addr)

d2010

  • Bull Frog
  • Posts: 322
Re: How to improve the computing speed of MD5
« Reply #15 on: October 21, 2022, 04:20:40 PM »
syz-md5.fas is implemented by calling the system api with pure lsp.
and this is great news, baitang36
well done
i think it brings lisp to a new level
Do you CormanLisp is more-fast than Visual-Lisp?
How to calculate the speed  of "CormanLisp"?

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8589
  • AKA Daniel
Re: How to improve the computing speed of MD5
« Reply #16 on: October 21, 2022, 05:50:17 PM »
syz-md5.fas is implemented by calling the system api with pure lsp.
and this is great news, baitang36
well done
i think it brings lisp to a new level
Do you CormanLisp is more-fast than Visual-Lisp?
How to calculate the speed  of "CormanLisp"?

Extending Autolisp with ObjextARX, in a general sense, is flawed because AutoCAD tries to translate some data structures.
For example, if you send  '(220 float float float)  from Autolisp to Arx, AutoCAD will crash with a LsAdsInvoke Internal Error.
'(10 float float float)  and you just a get a point on the arx side, so there's overhead of converting data structures back

However, like in this case, a MD5 routine, where the arguments are predetermined, Extending Autolisp with ObjextARX can be useful

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8589
  • AKA Daniel
Re: How to improve the computing speed of MD5
« Reply #17 on: October 21, 2022, 11:34:40 PM »
Someone can test the performance of this. compiled for BricsCADv22 and autocad 2021-23 and the source is included

it's not optimal because I'm using all 8 bytes for each element, this list would only need 2 bytes for each.
it should generate a hash for any list.  :mrgreen:
« Last Edit: October 21, 2022, 11:51:37 PM by It's Alive! »

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8589
  • AKA Daniel
Re: How to improve the computing speed of MD5
« Reply #18 on: October 22, 2022, 12:04:46 AM »
this one is slightly better, I didn't include NIL or T.. is it needed?


Code - C: [Select]
  1.     union uINT16Data
  2.     {
  3.         int16_t val;
  4.         BYTE buf[2];
  5.     };
  6.     union uINT32Data
  7.     {
  8.         int32_t val;
  9.         BYTE buf[4];
  10.     };
  11.     union uINT64Data
  12.     {
  13.         int64_t val;
  14.         BYTE buf[8];
  15.     };
  16.     union uDOUBLEData
  17.     {
  18.         double_t val;
  19.         BYTE buf[8];
  20.     };
  21.  
  22.     //(ads_md5 '(39 50 50 5 33 36 37 37 25 29 31 32 32 1 19 24 24 25 28 29 13 14 14 17 18 21 21 23 12))
  23.     static int ads_ads_md5(void)
  24.     {
  25.         constexpr size_t MD5LEN = 16;
  26.         constexpr CHAR rgbDigits[] = "0123456789abcdef";
  27.  
  28.         HCRYPTPROV hProv = 0;
  29.         HCRYPTHASH hHash = 0;
  30.         BYTE rgbHash[MD5LEN];
  31.         DWORD cbHash = 0, dwStatus = 0;
  32.         std::vector<BYTE> bytes;
  33.  
  34.         unique_resbuf_ptr pArgs(acedGetArgs());
  35.  
  36.         for (resbuf* pTail = pArgs.get(); pTail != nullptr; pTail = pTail->rbnext)
  37.         {
  38.             switch (pTail->restype)
  39.             {
  40.                 case RTSHORT:
  41.                 {
  42.                     uINT16Data dbl{ pTail->resval.rint };
  43.                     for (size_t idx = 0; idx < 2; idx++)
  44.                         bytes.push_back(dbl.buf[idx]);
  45.                 }
  46.                 break;
  47.                 case RTLONG:
  48.                 {
  49.                     uINT32Data dbl{ pTail->resval.rlong };
  50.                     for (size_t idx = 0; idx < 4; idx++)
  51.                         bytes.push_back(dbl.buf[idx]);
  52.                 }
  53.                 break;
  54.                 case RTREAL:
  55.                 {
  56.                     uDOUBLEData dbl{ pTail->resval.rreal };
  57.                     for (size_t idx = 0; idx < 8; idx++)
  58.                         bytes.push_back(dbl.buf[idx]);
  59.                 }
  60.                 break;
  61.                 case RTPOINT:
  62.                 {
  63.                     {
  64.                         uDOUBLEData dbl{ pTail->resval.rpoint[0] };
  65.                         for (size_t idx = 0; idx < 8; idx++)
  66.                             bytes.push_back(dbl.buf[idx]);
  67.                     }
  68.                     {
  69.                         uDOUBLEData dbl{ pTail->resval.rpoint[1] };
  70.                         for (size_t idx = 0; idx < 8; idx++)
  71.                             bytes.push_back(dbl.buf[idx]);
  72.                     }
  73.                 }
  74.                 break;
  75.                 case RT3DPOINT:
  76.                 {
  77.                     {
  78.                         uDOUBLEData dbl{ pTail->resval.rpoint[0] };
  79.                         for (size_t idx = 0; idx < 8; idx++)
  80.                             bytes.push_back(dbl.buf[idx]);
  81.                     }
  82.                     {
  83.                         uDOUBLEData dbl{ pTail->resval.rpoint[1] };
  84.                         for (size_t idx = 0; idx < 8; idx++)
  85.                             bytes.push_back(dbl.buf[idx]);
  86.                     }
  87.                     {
  88.                         uDOUBLEData dbl{ pTail->resval.rpoint[2] };
  89.                         for (size_t idx = 0; idx < 8; idx++)
  90.                             bytes.push_back(dbl.buf[idx]);
  91.                     }
  92.                 }
  93.                 break;
  94.                 case RTSTR:
  95.                 {
  96.                     std::string str = wstr_to_utf8(pTail->resval.rstring);
  97.                     for (auto ch : str)
  98.                         bytes.push_back(ch);
  99.                 }
  100.                 break;
  101.                 default:
  102.                     break;
  103.             }
  104.         }
  105.         if (!CryptAcquireContext(&hProv, NULL, NULL, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT))
  106.         {
  107.             dwStatus = GetLastError();
  108.             acutPrintf(L"CryptAcquireContext failed: %d\n", dwStatus);
  109.             acedRetNil();
  110.             return (RSRSLT);
  111.         }
  112.  
  113.         if (!CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash))
  114.         {
  115.             dwStatus = GetLastError();
  116.             acutPrintf(L"CryptAcquireContext failed: %d\n", dwStatus);
  117.             acedRetNil();
  118.             return (RSRSLT);
  119.         }
  120.  
  121.         if (!CryptHashData(hHash, bytes.data(), bytes.size(), 0))
  122.         {
  123.             dwStatus = GetLastError();
  124.             acutPrintf(L"CryptHashData failed: %d\n", dwStatus);
  125.             CryptReleaseContext(hProv, 0);
  126.             CryptDestroyHash(hHash);
  127.             acedRetNil();
  128.             return dwStatus;
  129.         }
  130.  
  131.         CString hash;
  132.         cbHash = MD5LEN;
  133.         if (CryptGetHashParam(hHash, HP_HASHVAL, rgbHash, &cbHash, 0))
  134.         {
  135.             for (DWORD i = 0; i < cbHash; i++)
  136.             {
  137.                 hash += (wchar_t(rgbDigits[rgbHash[i] >> 4]));
  138.                 hash += (wchar_t(rgbDigits[rgbHash[i] & 0xf]));
  139.             }
  140.             acedRetStr(hash);
  141.         }
  142.         else
  143.         {
  144.             dwStatus = GetLastError();
  145.             acutPrintf(L"CryptGetHashParam failed: %d\n", dwStatus);
  146.         }
  147.         CryptDestroyHash(hHash);
  148.         CryptReleaseContext(hProv, 0);
  149.         return (RSRSLT);
  150.     }
  151.  

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8589
  • AKA Daniel
Re: How to improve the computing speed of MD5
« Reply #19 on: October 22, 2022, 12:18:50 AM »
Actually, this could be handy for creating a hash map of lists... maybe

Code - Text: [Select]
  1. (map of lists
  2.   '("bada286cb105994eb4b2426308dee64d" 39 50 50 5 33 36 37 37 25 29 31 32 32 1 19 24 24 25 28 29 13 14 14 17 18 21 21 23 12)
  3.   '("100d9e9bc0ef4990807d95c7faebaba0" 39 50 50 5 33 36 37 37 25 29 31 32 32 1 19 24 24 25 28 29 13 14 14 17 18 21 21 23 32)
  4.    ...
  5. )

Lee Mac

  • Seagull
  • Posts: 12899
  • London, England
Re: How to improve the computing speed of MD5
« Reply #20 on: October 22, 2022, 03:05:31 PM »
Here's an incredibly slow one... mostly due to the fact that I had to represent 32-bit unsigned integers as lists of binary digits (as AutoLISP only has signed 32-bit integers), and then recreate list equivalents to the lsh & boole functions (which can only operate on 32-bit signed integers) to perform the various bitwise operations...  :lol:

It's Alive!

  • Retired
  • Needs a day job
  • Posts: 8589
  • AKA Daniel
Re: How to improve the computing speed of MD5
« Reply #21 on: October 22, 2022, 06:13:38 PM »
Awesome code LM!

Lee Mac

  • Seagull
  • Posts: 12899
  • London, England
Re: How to improve the computing speed of MD5
« Reply #22 on: October 23, 2022, 05:30:11 AM »
Thanks Daniel  :-)

xdcad

  • Bull Frog
  • Posts: 424
Re: How to improve the computing speed of MD5
« Reply #23 on: November 22, 2023, 06:19:10 PM »
syz-md5.fas is implemented by calling the system api with pure lsp
when this possibility will be available for the normal people ? :-)

(xdrx-file-md5hash fn)
(xdrx-string-md5hash "abcdefg")

Command: (xdrx-string-md5hash "abcdefg")
"7ac66c0f148de9519b8bd264312c4d64"
The code I wrote uses XDRX-API,which can be downloaded from github.com and is updated at any time.
===================================
[XDrx-Sub Forum]
https://www.theswamp.org/index.php?board=78.0
https://github.com/xdcad/XDrx-API
http://bbs.xdcad.net