Troubleshooting Pilot Balloon Data Acquisition in Real Time
  1. Problem: When I go to www.nssl.noaa.gov/projects/pacs/realtlist.shtml, the current date/time does not appear in the list of available data.
Program runam may not be running. This program is scheduled to run at 7:30 in the morning and it runs continuously until 10:00 p.m.; sometimes, failures prevent runam from starting at the scheduled hour.

You can issue the following command to see if runam is in the list of active processes in vortex for user jmurillo:

ps -edaf | grep jmurillo

Two lines similar to the following should appear if runam is running:

jmurillo 2539105 2537768 0 07:30:00 ? 0:20 /usr/bin/perl -w ./s2washam

jmurillo 2537768 2360186 0 07:30:00 ? 0:00 /usr/bin/tcsh /usr/people/jmurillo/ops/runam
 
 

a) If the program is not running, perform the following steps to start runam:

b) If runam is running but the current date/time is still not shown in the list, go to Page 2.

 
 
  1. Problem: The program runam is running but the current date/time does not appear in the list at www.nssl.noaa.gov/projects/pacs/realtlist.shtml,

    2.  

     
     

    If runam is running, then the problem might be with the script that moves (ftp) data from sounding.nssl.noaa.gov to vortex.protect.nssl. The script's name is Frmsndg and it is scheduled to run about every half an hour, but sometimes problems in vortex prevent the cron from starting the script.

    How do I know Frmsndg is being run as scheduled (every half an hour)?

    - Log in to vortex using pacs account (username pacs) and issue the following command to verify the times the script must be launched at:

    crontab -l

    Then issue one the following commands and look at the dates and hours the script Frmsndg has been running:

    cat FTPLOG1 (in the morning)

    or

    cat FTPLOG2 (for 00z soundings)

    If Frmsndg did not run within the past hour (or hours), this is a problem that has to be reported by email to Steve.Fletcher@noaa.gov or by submitting a report to dirt.nssl.noaa.gov.

    What can I do in the mean time?

    Frmsndg can be run at any time by issuing one of the following commands:

    Frmsndg 1 (for ~12z soundings)

    Frmsndg 2 (for ~00z soundings)

    You need to do this every time new data arrives to soundings.nssl.noaa.gov.

  2. Problem: runam is running and Frmsndg seems to be working just fine, but the current date/time is still not shown in the list of dates with data received in real time (www.nssl.noaa.gov/projects/pacs/realtlist.shtml)
The reason for this is that none of the PACS-SONET stations is sending data in real time because of lack of observations or because they are unable to connect to soundings.nssl.noaa.gov. Look for observations being sent by email and transfer those to vortex (user pacs, directory /usr/local/ftp/incoming/stationname/, where stationname is apure, asuncion, cam, carmen, etc.) Also make sure that soundings is up and running by logging in to that machine.

Another possible cause of this problem could be a failure in the main computer (namely: the "no space left in disk", or other failure). Report problems like this one to Steve.Fletcher@noaa.gov or submit a report to dirt.nssl.noaa.gov.

4. Problem: No graphic products. There is data for the current date/time, but the map with the stations and heights for each sounding is not displayed. The page typically looks like this in Netscape®:


 
 

There are at least two possible reasons for this problem.

  1. The program that generates the graphic products is not running. This program's name is runmaps; it is launched by a cron job every half an hour, beginning at 7:30 in the morning. Sometimes, failures prevent runmaps from starting at the scheduled hour.


    2. What can I do in the mean time? The program can be run at anytime by logging in to vortex as jmurillo (username jmurillo) and issuing the following commands:

    cd /usr/people/jmurillo/ops

    touch ok.new

    runmaps

    (Don't close the window while runmaps is running.)

    If you succeed generating the graphics using this method, it means the cron jobs are not running as expected and the problem should be reported to the ITS personnel. If you don't succeed, and you only get the message "version previa de gpcolsemi en ejecucion", go to Page 5 .

  2. The programs that generates the graphic products have been running for several minutes (or hours). See the next page for an explanation on how to diagnose and fix this problem.
  1. Problem: There are no graphic products for the current date/time and I already tried to run runmaps myself as explained in 4(a) without success.
The most frequent reason for this problem is related to my computer (Sonet) and software that must be running on it (MacX). The Gempak meteorological package is being used to generate graphic files and it requires an X Window display server application to be installed and run on the issuing computer in order to generate graphic files in batch mode. Therefore, if Sonet or the MacX application crashes, the graphics won't be created or updated.
  1. If Sonet didn't crash, or if you get the message "version previa de gpcolsemi en ejecucion" when trying the procedure in 4(a), then Mac X may have crashed. Find out issuing the following command:


    2. ps -edaf | grep jmurillo

    You will see several lines with information on the processes associated to different Gempak programs, namely snmap, gpcolor, gplt, gf, etc.:

    jmurillo 2123100 2123612 0 21:59:45 pts/0 0:00 gf

    jmurillo 2124369 2105194 0 21:55:04 pts/0 0:00 /bin/csh ./gpcolsemi 02052800 20020528

    jmurillo 2105194 2112383 0 21:55:04 pts/0 0:00 /bin/sh ./gpcolsemi.cmd

    jmurillo 2078418 2077915 0 07:30:01 ? 1:22 /usr/bin/perl -w ./s2washam

    jmurillo 2112383 2104996 0 21:55:04 pts/0 0:00 /usr/bin/tcsh runmaps

    jmurillo 2121184 2124369 0 22:00:19 pts/0 0:00 gpend

    the most important thing to check about the above listing is the times at which the Gempak programs were initiated. If they are about one hour old or more, it means Max X application failed.

    Is there an easy way to fix this problem?

    Of course. Log in to vortex as user jmurillo (Username: jmurillo) and kill Gempak-related processes in the following order : runmaps, gpcolsemi, gpcolor, snmap, gpend, gplt, gf. You can easily accomplish this by issuing the following command:

    /usr/people/jmurillo/bin/cleangp

    The command will return a list of the active processes. This should be a short list with none of the Gempak programs in it.

    Finally, restart Sonet. Mac X application will open automatically at start. Try doing the procedure in 4(a).

  2. If Sonet crashed, restart it. Mac X application will be initiated automatically. Then try the procedure explained in Page 4(a) above.