• Created by Unknown User (morgan), last modified by Unknown User (fackler) on Nov 11, 2023

BOX - beginning of expedition
EOX - end of expedition

The outgoing programmer(s) is(are) responsible for all of the end-of-expedition (EOX) activities.

This expedition is ending.

Make sure we are getting as complete a copy of the data to be warehoused on shore as we can.

This document does not recapitulate all items on the checklist. If an item is in need of elaboration, please add to it.

0. Negotiate a time window

Work with the LO, ALOs, EPM, MCS, laboratory Technical Staff. Three weeks before the end of the expedition you should have an (increasingly clear) idea how and when these activities will transpire.

  • When will the last core be on deck?
  • When will we be in port?
  • When do we need to be done collecting data for a given laboratory?
  • Has all the data content that is required to be warehoused been stored?
      uploaded to the database
      copied to data1 OR
      copied to uservol (workspace toward creation of EXPEDITION PROCEEDINGS)
  • Are you aware of the varieties of data that have been collected?
    A review of data1 is helpful
    Though we presently do not fill this out, a data disposition spreadsheet/form is available in s:\data1 that  exercises this concern.

If not done, obtain estimates when:

  • routine core flow will be complete
  • data is uploaded and copied to data1
  • routine sampling will be complete

Pacing items are typically XRD (long powder prep times), MAD (24-hour oven drying time), SRM (typically last track run), some chemistry processes have lengthy sample prep and run procedures.

It is Ok to ask repeatedly. It is an exercise in planning and preparation which helps us collectively focus and prioritize the tasks to be done.

1. Deliver the EOX tech report

Deliver the end-of-expedition technical report to the (place in Confluence specified by the) LO.
Please post a PDF export of it to jrso-developer on Slack.

2. Establish when the final database snapshot will be taken

Recapitulation of 0. Continue talking to the LOs, ALOs, EPM, MCS, and laboratory Technical Staff to clarify this timing.

  • It takes 20 minutes to run a full database export (for 61 GiB of data)--but we want it to be as complete as possible within the constraints of expedition activities.
  • The MCS routinely stage expedition content for convenience of copying via Science Office and User Room workstations. Timing of that activity is a common communication at the end of the expedition.

3. Spot check data copy efforts

Recapitulation of 0, 2. Be an additional set of eyes and ears.

What data didn’t get into the database? What remains to be staged on data1 or uservol?
As-needed, provide support to get data in or find an appropriate staging location for it. 

  • Spot check instrument host data folders. Are they getting over to data1?
  • If technical staff are new--are they aware of the scope of concerns for getting data back home?
  • If technical staff are old hands--learn about what they do and why.

Often there is both expedition-specific and individual variance. Much room for improvement in the current process.

4. Honor moratorium

Once content is uploaded to LIMS, and raw files have gone to data1, and routine backup cycles have occurred: expedition-specific data may (in-general) be cleaned off of instrument hosts and workstations.

These activities are the purview of laboratory technical staff. Often assistance is welcome.

There is precedent for this activity to be pushed to the (fresh) oncoming crew. Totally depending on timing of last core on deck and timing of arrival in port.

5. Conduct EOX database backup

Make the database read only

Run from PuTTY, Windows Terminal, Powershell, Hyperterminal.
You can run this step from SQL Developer too, but you'll need the command-line at the ODA for the data pump export command. If the oraenv script does not show LIMSJR1 as the default, contact the DBA (Sunil).

ssh oracle@k1.ship.iodp.tamu.edu

. oraenv
ORACLE_SID = [LIMSJR1] ? <Enter>

sql you_dba

sql>
alter tablespace labware read only;
exit

[oracle]$

Export the database content

cd /backup/export
ls -l
time expdp you_dba directory=dmpdir full=y logfile=limsjr_full-###-export.log dumpfile=limsjr_full-###.dmpdp

Make the database read-write again

sql you_dba
Password:

sql>
alter tablespace labware read write;
exit

[oracle]$

Test the export file

Read the limsjr_full-###-export.log in k1:/backup/export.

  • Are there any errors? No is good. If errors, find cause, maybe re-run.
  • Review the number of samples for LIMS.SAMPLE, LIMS.TEST, LIMS.RESULT, OPS.RIS_DATA.

Log into the SHIPTEST instance. Run these commands to confirm ability to restore content from the file generated.

ssh oracle@k2.ship.iodp.tamu.edu

. oraenv
ORACLE_SID = [SHIPTEST] ? <Enter>
[oracle]$

time impdp you_shiptest_dba directory=dmpdir logfile=exp###-test-export.log dumpfile=limsjr_full-###.dmpdp tables=lims.sample,lims.test remap_schema=lims:transfer remap_table=lims.sample:xxx_sample,lims.test:xxx_test exclude=trigger table_exists_action=replace

time impdp you_shiptest_dba directory=dmpdir logfile=exp###-test-export.log dumpfile=limsjr_full-###.dmpdp schemas=geodp### remap_schema=geodp###:geodp###_review

Use SQL Developer to review the content of the SHIPTEST TRANSFER.XXX_SAMPLE, XXX_TEST tables. Repeat for the restored GEODP###_REVIEW schema. Do not cleanup. Leave it for the oncoming crew to review.

Notify everyone that its done

Post the export log file on Slack jrso-developer. Note the size of the export file.

We will not zip it, nor send it over the wire. If needed to achieve earlier emplacement of content on shore, the person tagged with conducting the EOX Shore data processing will conduct those activities.

Email it to

  • MCS - jr_mcs@ship.iodp.tamu.edu
  • IT_OPS - it_ops@iodp.tamu.edu
  • Data Librarian - database@iodp.tamu.edu
  • programmers - programmers@iodp.tamu.edu
  • DBAs - sunil@rajaconsultancy.com, murli@rajaconsultancy.com

In the email, note the size of the file and its location--if we're putting it in the right place, there's no need for the MCS to move it.

Speak with the MCS. They will pick up the data from here.

oemjr:/backup/export
    limsjr*.dmpdp
    limsjr*.log


The MCS pick up the full export from  oemjr:/backup/export.

Enquire the MCS. Ensure the (above) database content and the (below) ASMAN content are being taken to media for transport to HQ.

olympus:/asman/catalogs
olympus:/asman/liveimages

Take a copy of the export file

Login into the Oracle account at the ODA using WinSCP. Bring down a copy of the export file and its log to

  • your laptop--or other transfer device; and
  • stage a copy in \\novarupta\vol3\dml\oracle-data\
    for future local reference OR until we have to clean/preen for space.

6. Provide courier services

The backup tapes created by the MCS return to HQ as soon as possible. If your travel itinerary supports that goal, let them know you can provide courier services.

  • No labels