That seems promising. Assuming you're working with Ryan's data, if you can successfully filter it, is it possible to send him the msieve output files so he doesn't have to redo the work?

If frmky is able to form a matrix, either he or I can run the LA. I'd also be interested to get the msieve patch so I can try the filtering step myself.

1 Attachment(s)
[CODE]Sat Feb 6 23:19:11 2021 commencing 2way merge
Sat Feb 6 23:28:47 2021 reduce to 334640603 relation sets and 328431256 unique ideals Sat Feb 6 23:28:47 2021 commencing full merge Sun Feb 7 04:51:15 2021 memory use: 38947.4 MB Sun Feb 7 04:52:10 2021 found 147393213 cycles, need 147295456 Sun Feb 7 04:53:37 2021 weight of 147295456 cycles is about 19148588285 (130.00/cycle) Sun Feb 7 04:53:37 2021 distribution of cycle lengths: Sun Feb 7 04:53:37 2021 1 relations: 11327174 Sun Feb 7 04:53:37 2021 2 relations: 10934036 Sun Feb 7 04:53:37 2021 3 relations: 11762737 Sun Feb 7 04:53:37 2021 4 relations: 11269419 Sun Feb 7 04:53:37 2021 5 relations: 10840433 Sun Feb 7 04:53:37 2021 6 relations: 10270300 Sun Feb 7 04:53:37 2021 7 relations: 9536741 Sun Feb 7 04:53:37 2021 8 relations: 8923986 Sun Feb 7 04:53:37 2021 9 relations: 8177647 Sun Feb 7 04:53:37 2021 10+ relations: 54252983 Sun Feb 7 04:53:37 2021 heaviest cycle: 28 relations Sun Feb 7 04:54:54 2021 commencing cycle optimization Sun Feb 7 05:08:34 2021 start with 1239155413 relations Sun Feb 7 07:05:08 2021 pruned 42215931 relations Sun Feb 7 07:05:10 2021 memory use: 34774.5 MB Sun Feb 7 07:05:11 2021 distribution of cycle lengths: Sun Feb 7 07:05:11 2021 1 relations: 11327174 Sun Feb 7 07:05:11 2021 2 relations: 11209087 Sun Feb 7 07:05:11 2021 3 relations: 12224988 Sun Feb 7 07:05:11 2021 4 relations: 11643676 Sun Feb 7 07:05:11 2021 5 relations: 11239934 Sun Feb 7 07:05:11 2021 6 relations: 10573210 Sun Feb 7 07:05:11 2021 7 relations: 9830741 Sun Feb 7 07:05:11 2021 8 relations: 9126063 Sun Feb 7 07:05:11 2021 9 relations: 8350544 Sun Feb 7 07:05:11 2021 10+ relations: 51770039 Sun Feb 7 07:05:11 2021 heaviest cycle: 28 relations Sun Feb 7 07:13:57 2021 RelProcTime: 199990 Sun Feb 7 07:13:57 2021 elapsed time 55:33:18[/CODE] 147.3M matrix! That would take my little 8 nodes about 2.5 months to solve! Patch is attached. Edit: Had to add .txt to the end to get the forum to accept it. 
[QUOTE=frmky;571096]147.3M matrix! That would take my little 8 nodes about 2.5 months to solve![/QUOTE]
Any chance you'd be able to run it? :smile: Otherwise, I'll test out the patch and can try sieving some more. 
It's now in the official msieve SVN, so no need for the patch file. Just grab the latest code.

[CODE]commencing linear algebra
read 147295456 cycles cycles contain 531217644 unique relations read 531217644 relations using 8 quadratic characters above 4294917295 building initial matrix memory use: 68478.7 MB read 147295456 cycles matrix is 147295267 x 147295456 (74519.1 MB) with weight 19922810683 (135.26/col) sparse part has weight 18061791563 (122.62/col) filtering completed in 2 passes matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col) sparse part has weight 18061766673 (122.63/col) matrix starts at (0, 0) matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col) sparse part has weight 18061766673 (122.63/col) saving the first 48 matrix rows for later matrix includes 64 packed rows matrix is 147292392 x 147292629 (71275.3 MB) with weight 17993852562 (122.16/col) sparse part has weight 17211463600 (116.85/col) using block size 8192 and superblock size 1966080 for processor cache size 20480 kB commencing Lanczos iteration (16 threads) memory use: 73902.4 MB linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)[/CODE] I'll see you in 4 years... :lol: 
[QUOTE=frmky;571129][CODE]
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)[/CODE] I'll see you in 4 years... :lol:[/QUOTE] Holy {censored}!!!! I mean, wow. 147M dimensions.... But the ETA seems off shouldn't this take around 4x as long as 73M dimensions? I solved one that size for 2330L in about 3100hr on a single 10core Ivy Bridge socket, and around 2000hr when run naively (without MPI) on both 10core sockets. I'm not volunteering for ~8000 machinehours, but it seems an 8socket minicluster should be around 10x faster than this! Thanks for finding and fixing this bug I'm happy that we're likely to have msieve at our disposal for jobs as big as we can solve matrices for! 
I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.
I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE. 
[QUOTE=frmky;571140]I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.
I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.[/QUOTE] If you're able to run this on your cluster, that would be greatly appreciated; if not, I also understand. Meanwhile I will continue sieving to see if I can form something resembling a reasonable matrix. 
Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64core workstation:
[CODE]commencing linear algebra read 147295123 cycles cycles contain 531218272 unique relations read 531218272 relations using 8 quadratic characters above 4294917295 building initial matrix memory use: 68478.7 MB read 147295123 cycles matrix is 147294934 x 147295123 (74519.4 MB) with weight 19922890703 (135.26/col) sparse part has weight 18061865192 (122.62/col) filtering completed in 2 passes matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col) sparse part has weight 18061839994 (122.63/col) matrix starts at (0, 0) matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col) sparse part has weight 18061839994 (122.63/col) saving the first 240 matrix rows for later matrix includes 256 packed rows matrix is 147291864 x 147292281 (69444.9 MB) with weight 16505103766 (112.06/col) sparse part has weight 15847879420 (107.59/col) using block size 8192 and superblock size 608256 for processor cache size 25344 kB commencing Lanczos iteration (64 threads) memory use: 98165.2 MB linear algebra at 0.0%, ETA 4273h 0m292281 dimensions (0.0%, ETA 4273h 0m) checkpointing every 40000 dimensions292281 dimensions (0.0%, ETA 4287h45m) linear algebra completed 81656 of 147292281 dimensions (0.1%, ETA 4336h46m)[/CODE] Unfortunately I don't think I can tie up this machine for ~half year, but hopefully some more sieving can bring down the matrix size a bit. 
[QUOTE=ryanp;571253]Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64core workstation:[/QUOTE]
Is this using MPI? On a multicpu system that should help. 
All times are UTC. The time now is 20:15. 
Powered by vBulletin® Version 3.8.11
Copyright ©2000  2021, Jelsoft Enterprises Ltd.