BOX - beginning of expedition
EOX - end of expedition

The outgoing programmer(s) is(are) responsible for all of the end-of-expedition (EOX) activities.

This expedition is ending.

Make sure we are getting as complete a copy of the data to be warehoused on shore as we can.goal is to ensure a good copy of the data gets back to HQ.

Please read this entire document and its companion. Expect to take 3 to 4 hours to complete this exercise. (I keep saying this only takes 15 minutes. But that really is just the individual database dump processes.) Do not hurry. If you are "fried"--work together. 

0. Readiness assessment.

Work the LO, ALOs, laboratory technical staff. Three weeks before the end of the expedition you should have an (increasingly clear) idea how and when these activities will transpire.

• When will the last core be on deck?
• When will we be in port?
• When do we need to be done collecting data for a given laboratory?
• Has all the data content that is required to be warehoused been stored?
    uploaded to the database
    copied to data1 OR
    copied to uservol (workspace toward creation of EXPEDITION PROCEEDINGS)
  
• Are you aware of the varieties of data that have been collected?
  A review of data1 is helpful
  Though we presently do not fill this out, a data disposition spreadsheet/form is available in s:\data1 that  exercises this concern.
  

If not done, obtain estimates when:

• routine core flow will be complete
• data is uploaded and copied to data1
• routine sampling will be complete

Pacing items are typically XRD (long powder prep times), MAD (24-hour oven drying time), SRM (typically last track run), some chemistry processes have lengthy sample prep and run procedures.

It is Ok to ask repeatedly. It is an exercise in planning and preparation which helps us collectively focus and prioritize the tasks to be done.

1. Deliver the EOX tech report

Deliver the end-of-expedition technical report to the (place in Confluence specified by the) LO.
Please post a PDF export of it to jrso-developer on Slack.

See the Confluence Lab Notebook space for prior examples of the development technical report. Keeping a daily activity log throughout the expedition aids in the timely preparation of the report.

The report can be the basis of cross-over discussions. It is also a records of expedition activities and concerns that can provide context to future users of IODP data.

2. Establish when the final database snapshot will be taken

Recapitulation of 0. Continue talking to the LOs, CoChiefs, MCS, and laboratory Technical Staff to clarify this timing.

• It takes 20 minutes to run a full database export (for 61 GiB of data)--but we want it to be as complete as possible within the constraints of expedition activities.
• The MCS routinely stage expedition content for convenience of copying via Science Office and User Room workstations. Timing of that activity is a common communication.

3. Circulate through the labs

Recapitulation of 0, 2. Be an additional set of eyes and ears.

What data didn’t get into the database? What remains to be staged on data1 or uservol?
As-needed, provide support to get data in or find an appropriate staging location for it. 

• Spot check instrument host data folders. Are they getting over to data1?
• If technical staff are new--are they aware of the scope of concerns for getting data back home?
• If technical staff are old hands--learn about what they do and why.

Often there is both expedition-specific and individual variance. Much room for improvement in the current process.

4. Honor moratorium

This is your concern too, but management of specific instrument hosts is the purview of lab technical staff. This task occurs in both EOX and BOX processes.

If end-of-expedition activities are harried and crunched--don't do this step. Leave the data for the (fresh) oncoming developers and technical staff to accomplish. 

Once content is uploaded to LIMS, and raw files have gone to data1, and routine backup cycles have occurred: expedition-specific data may (in-general) be cleaned off of instrument hosts and workstations.

It is good practice to assist technical staff in this activity: both to raise awareness of its complexity, and to consider opportunities for automation and improvement. At busy times it can help reduce individual workload. Well managed labs will have staging areas where expedition-specific data is placed so it can be readily managed in this context.

Good practice to confer with technical staff that manage this for their labs. There is variance between crews as to how and when these procedures are carried out. It is possible to do these activities on a site-by-site basis rather than just EOX.

5. Conduct EOX database backup

6. Provide courier services

The backup tapes created by the MCS return to HQ as soon as possible. If your travel itinerary supports that goal, let them know you can provide courier services.

1. Notify everyone. Database snapshot in progress.

• Notify jr_developer and the shipboard technical staff that the EOX database backup is in progress.
• The database is available for reports, but content cannot be changed while this activity is in progress.

2. Make the database read only.

Connect to the BUILD box. Execute all of the following steps from that system.


Run PuTTY.
Connect to oracle@k1.ship.iodp.tamu.edu
From the prompt run this command, supply the oracle user's credentials for k1

At the SQL> prompt, run the last command.
Stop. Leave this window open for use below.

Technical note. This command prevents write access to any table content that resides in the storage facility under the "labware" table space. In our database implementation, this happens to be all tables owned by the LIMS schema. Use of Drill Report ("ops" schema) is not affected. Aspects of DescLogik software configuration are also still open to maintenance, though use of DescLogik itself is locked out by this change.

3. Full snapshot as oracle@k1.

Run another PuTTY session.
Connect to oracle@k1.ship.iodp.tamu.edu
From the prompt run these commands--change the expedition and yourname  (will require logging in as a DBA User see PWSAFE) - - - JON Put Your Password in "QUOTES"

NOTE: While performing the EOX for expedition 384, it was discovered that the path represented by "data_pump_dir" did not exist.  You can discover what the data_pump_dir is supposed to be using this query (logged in as your DBA account):

select * from dba_directories;

The results have a lot of stuff that's not useful, but the DATA_PUMP_DIR should be there.  Currently it is /backup/LIMSJR/dpdump.  Note that "backup" is singular (it used to be plural) and there is apparently some desire to change this, so keep that in mind.  If you find that the folder does not exist, the command in this step will fail and you may need to get an MCS or a DBA on shore to help create it.

You can also try:

create directory data_pump_dir as '/backup/export';
grant read, write on directory data_pump_dir to transfer, system;


10/5/2021: Folder did not exist, as above, but MCS suggested I try creating it myself (in Putty, logged in as Oracle) using mkdir commands.  That initially seemed to work, but it broke some kind of symbolic or virtual link that caused the file to be written to the wrong physical location (with reduced disk space), so not recommended.

Statistics. Typical durations for the last step from previous expeditions. 344S: 16 min 23 sec. 345: 13 min 54 sec. 346: 18 min 46 sec. 341S: 11 min 42 sec. 351: 12 min 9 sec; 360: 16 min 57 sec; 363: 3 min 21 sec; 367: 3 min 38 sec.

ODA for 3 expeditions of data (11.1 GiB): 4 min 17 sec.
396: 6 min 54 sec

4. Restore database read/write capability.

From the prompt [oracle@k1 ~]$

run these command, supply the appropriate credential

Stop. Leave this window open for use below.

5. Compress the exports for transport. Cleanup. 

From the prompt [oracle@k1 ~]$
run these commands

Cut and paste the digest keys and file listing into a notification email. Enables remote users to verify the integrity of the files.

           Please log these in a txt file in the \AD\support\'expedition named sub directory'\ see other expeditions for example.

Statistics. 367: 22 min to compress both files (~5 GiB to < 1 GiB); 345: 22 min 53 sec for bzip step. 351: full 137 min 46 for ~8 GiB content.  360: 67 min 27 sec for ~5 GiB file. 45-60 min for high recovery expeditions.

Exp 396: we had trouble initially because we ran out of disk space.  This was apparently due to my creating the /backup/LIMSJR/dpdump folder under the Oracle account.  This broke some kind of link-up that gives the backup folder lots of additional drive space.  Don't do that.

Need to test DMPDP.
Need to post log results.

6. Notify everyone. Database snapshot process is complete.

 Notify jr_developer and expedition technical staff that the process is done.

Speak with the MCS. They will pick up the data from here.

The MCS pick up the full export from  oemjr:/backup/export.

Inquire with the MCS. Ensure the (above) database content and the (below) ASMAN content are being taken to media for transport to HQ.

1. Establish with MCS, LOs, CoChiefs when the final database snapshot will be taken for EOX.
2. Circulate through the labs. 
3. Honor moratorium. Once content is uploaded to LIMS, and raw files have gone to data1, expedition-specific data may be cleaned off of instrument hosts and workstations. Good practice to confer with technical staff that manage this for their labs. There is variance between crews as to how these procedures are carried out.
4. Conduct end of expedition procedures for backing up the database.
5. Provide courier services if called upon to do so.
6. Confirm with the MCS what is to be included in the backup going to shore and that it does cover all the information you are aware of that should go to shore.
7. Ensure all your code changes are checked in.
8. Clean the development office. Assist in the general cleaning efforts. 
9. Be ready to move out. Your replacement will be here soon.
10. Provide assistance and information to oncoming developers.
    The goal of all oncoming personnel is to ensure the function of laboratory and computing systems to support the successful execution of the new expedition.
    Assist with those goals.
    They may not have sailed for quite a while, plan to
    1. Review your changes.
    2. Review the current state of the labs.
    3. Review the current state of deployed applications.