Welcome, Guest
Username: Password: Remember me

TOPIC: Artemis and direct solver in parallel (MUMPS)

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12581

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello,

It seems that Artemis is lacking a direct solver in parallel installation. Routine LECDON stops the computation if a direct solver is set in parallel.

In the past, MUMPS was used for that purpose (at least it looks that way from the perl installation scripts). Is MUMPS supported in the current version? I know that compiling MUMPS (and the pre-requisites) is not an easy task (even harder in windows) but I want to give it a try.

The question is: If I manage to compile MUMPS, will I be able to use it in artemis or MUMPS is deprecated from now on?

Best Regards,
Costas
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12599

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello,

Does somebody have an answer to this question? I do not want to put effort in compiling MUMPS if it is no longer supported. :(

Regards,
Costas
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12604

  • jmhervouet
  • jmhervouet's Avatar
Hello,

I am not currently at work, but MUMPS is still supported, and is called by subroutine solve.f in library bief, so it should be available for every module in the system.

Regards,

JMH
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12614

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello JMH,

Thank you for your reply. I have noticed the references in solve.f but in lecdon_artemis.f there is this part that stops the computation when the direct solver (option 8) is chosen with parallel:
! USE OF DIRECT SOLVEUR IS NOT POSSIBLE WITH PARALLELISM
!
      IF(NCSIZE.GT.1) THEN
        IF(SLVART%SLV.EQ.8) THEN
           IF(LNG.EQ.1) WRITE(LU,3004)
           IF(LNG.EQ.2) WRITE(LU,3005)
3004       FORMAT(1X,'AVEC PARALLELISME,',/,1X,
     &             'PAS DE SOLVEUR DIRECT',///)
3005       FORMAT(1X,'WITH PARALLELISM,',/,1X,
     &             'NO DIRECT SYSTEM SOLVER',///)
           CALL PLANTE(1)
I will try compiling MUMPS with mingw-w64, but when you are available, I could use you help with the issue above.

Best regards,
Costas
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12620

  • jmhervouet
  • jmhervouet's Avatar
Hello,

I am not sure, but it seems to me that MUMPS would be option 9 instead of 8. 8 is the Yale direct solver and does not work in parallel for sure.

Regards,

JMH
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12629

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello JM,

Yes, you are correct, option 9 is the MUMPS solver, but it is not documented in the manual.

I am using case BTWI to test my installation of MUMPS. So far I have managed to compile MUMPS and the pre-requisite libraries (ScaLapack, Scotch).

I have used Scalapack v2.0.2, Scotch v5.1.12 and MUMPS v4.1.0.

Here is my config file:
# _____                   ________________________________________
# ____/ windows generals /_______________________________________/
[general]
#
root:       C:\opentelemac\v6p3r2
version:    v6p3
language:   2
modules:    clean bief artemis -dredgesim
options:    parallel mpi
#
sfx_zip:    .zip
sfx_lib:    .lib
sfx_mod:    .mod
sfx_exe:    .exe
#
val_root:   <root>\examples
val_rank:   all
# also possible val_rank:   <3 >7 6
#
# _____                                           ____________________________________
# ____/ Windows gfortran parallel MSMPI (Corei7) /___________________________________/
[gfmsmpi-Corei7]
#
par_cmdexec:   <config>\partel.exe < PARTEL.PAR >> <partel.log>
#
#mpi_hosts:   
mpi_cmdexec: mpiexec.exe /wdir <wdir> /env MPICH_PORT_RANGE 10000,11000 /cores <ncsize> <exename>
#
cmd_obj:    x86_64-w64-mingw32-gfortran -march=corei7 -O2 -fopenmp -c -fno-range-check -ffixed-line-length-132 -fconvert=big-endian -frecord-marker=4 <mods> <incs> <f95name> -DHAVE_MPI -DHAVE_MUMPS
cmd_lib:    ar cru <libname> <objs>
cmd_exe:    x86_64-w64-mingw32-gfortran -static -march=corei7 -O2 -fopenmp -fconvert=big-endian -frecord-marker=4 -v -o <exename> <objs> -Xlinker --start-group <libs>
#--end-group
#
mods_all:   -I <config>
#
incs_parallel: -I c:\opentelemac\msmpi-4.2.4400.0-mingw-w64-x64\inc -I c:\opentelemac\mumps\include
libs_partel:	c:\opentelemac\libmetis\libmetis64b_5.0.2.a
libs_all:	c:\opentelemac\msmpi-4.2.4400.0-mingw-w64-x64\lib\libmsmpi.a -L c:\opentelemac\mumps\lib -ldmumps -lmumps_common -lpord -lscotch -lscotcherr -lscalapack
#
sfx_obj:    .o
#

Initial testing has produced this error:

File Attachment:

File Name: output.txt
File Size: 6 KB

Could this indicate the nature of my MUMPS installation problem?

Regards,
Costas
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12636

  • jmhervouet
  • jmhervouet's Avatar
Hello,

This seems an error occuring before calling MUMPS, when the 4-matrix system in Artemis is formatted into a single matrix problem for the direct solver. So Mumps installation is not (yet) guilty here. I'm afraid I'll have to look into this with my colleagues when we are all back (soon...).

Sorry for the inconvenience...

JMH
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12657

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello JM,

The error output I have previously uploaded occured when I had specified a 2 processor run. When I changed to 4 processors, I get a Segmentation fault error:
UB-ITERATION NUMBER :   1

 LINEAR SYSTEM SOLVING (SOLVE)

 SD_FABSG4: UNEXPECTED CASE

job aborted:
[ranks] message

[0] process exited without calling finalize

[1-3] terminated

---- error analysis -----

[0] on thor
out_btwi-par.exe ended prematurely and may have crashed. exit code 3

---- error analysis -----
_____________
runcode::main:
:
   |runCode: Fail to run
   |mpiexec.exe /wdir \\atlas\Company\DataDisk\Work\opentelemac\012_btwi\btwi-par.cas_2014-04-16-18h03min35s /env MPICH_PORT_RANGE 10000,11000 /cores 4 out_btwi-par.exe
   |~~~~~~~~~~~~~~~~~~
   |STOP 2
   |STOP 2
   |STOP 2
   |
   |Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
   |
   |Backtrace for this error:
   |~~~~~~~~~~~~~~~~~~
Does this provide more insight? Maybe it is the installation of MUMPS that is causing the trouble in the first place.

Regards,
Costas
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12681

  • jmhervouet
  • jmhervouet's Avatar
Hello,

We have just found with the direct solver (option 8) that in scalar mode, as soon as you have more than 2 millions of points in a mesh the memory allocations requested in sd_solve_1.f have sizes greater than the largest I4 integer, for example we have a case where NSP=1740130920 gives a 2*NSP which is negative because larger than HUGE(1)=2147483647. Could it be also your case ?

With best regards,

Jean-Michel Hervouet
The administrator has disabled public write access.

Artemis and direct solver in parallel (MUMPS) 10 years 7 months ago #12715

  • cyamin
  • cyamin's Avatar
  • OFFLINE
  • openTELEMAC Guru
  • Posts: 997
  • Thank you received: 234
Hello,

I am using case BTWI (20775 nodes) as a test and it runs without problem in direct solver (option 8) in scalar. The problem arises when using option 9 (MUMPS) in parallel. So it is more likely to be an issue with the MUMPS (and prerequisite) libraries.

I have some questions regarding the compilation of MUMPS library and its usage with Artemis:
  1. Is METIS optional or absolutely required?
  2. Does MUMPS require the scalar (METIS, SCOTCH) or parallel (PARMETIS, PTSCOTCH) versions of the prerequisite libraries?

Regards,
Costas
The administrator has disabled public write access.

The open TELEMAC-MASCARET template for Joomla!2.5, the HTML 4 version.