BOX - beginning of expedition
EOX - end of expedition
The outgoing programmer(s) is(are) responsible for all of the end-of-expedition (EOX) activities.
This expedition is ending.
Make sure we are getting as complete a copy of the data to be warehoused on shore as we can.
0. Negotiate a time window
Work with the LO, ALOs, EPM, MCS, laboratory Technical Staff. Three weeks before the end of the expedition you should have an (increasingly clear) idea how and when these activities will transpire.
- When will the last core be on deck?
- When will we be in port?
- When do we need to be done collecting data for a given laboratory?
- Has all the data content that is required to be warehoused been stored?
uploaded to the database
copied to data1 OR
copied to uservol (workspace toward creation of EXPEDITION PROCEEDINGS) - Are you aware of the varieties of data that have been collected?
A review of data1 is helpful
Though we presently do not fill this out, a data disposition spreadsheet/form is available in s:\data1 that exercises this concern.
If not done, obtain estimates when:
- routine core flow will be complete
- data is uploaded and copied to data1
- routine sampling will be complete
Pacing items are typically XRD (long powder prep times), MAD (24-hour oven drying time), SRM (typically last track run), some chemistry processes have lengthy sample prep and run procedures.
It is Ok to ask repeatedly. It is an exercise in planning and preparation which helps us collectively focus and prioritize the tasks to be done.
1. Deliver the EOX tech report
Deliver the end-of-expedition technical report to the (place in Confluence specified by the) LO.
Please post a PDF export of it to jrso-developer on Slack.
2. Establish when the final database snapshot will be taken
Recapitulation of 0. Continue talking to the LOs, ALOs, EPM, MCS, and laboratory Technical Staff to clarify this timing.
- It takes 20 minutes to run a full database export (for 61 GiB of data)--but we want it to be as complete as possible within the constraints of expedition activities.
- The MCS routinely stage expedition content for convenience of copying via Science Office and User Room workstations. Timing of that activity is a common communication at the end of the expedition.
3. Spot check data copy efforts
Recapitulation of 0, 2. Be an additional set of eyes and ears.
What data didn’t get into the database? What remains to be staged on data1 or uservol?
As-needed, provide support to get data in or find an appropriate staging location for it.
- Spot check instrument host data folders. Are they getting over to data1?
- If technical staff are new--are they aware of the scope of concerns for getting data back home?
- If technical staff are old hands--learn about what they do and why.
Often there is both expedition-specific and individual variance. Much room for improvement in the current process.
4. Honor moratorium
Once content is uploaded to LIMS, and raw files have gone to data1, and routine backup cycles have occurred: expedition-specific data may (in-general) be cleaned off of instrument hosts and workstations.
These activities are the purview of laboratory technical staff. Often assistance is welcome.
There is precedent for this activity to be pushed to the (fresh) oncoming crew. Totally depending on timing of last core on deck and timing of arrival in port.
5. Conduct EOX database backup
Make the database read only
Run from PuTTY, Windows Terminal, Powershell, Hyperterminal.
You can run this step from SQL Developer too, but you'll need the command-line at the ODA for the data pump export command.
ssh oracle@k1.ship.iodp.tamu.edu . oraenv ORACLE_SID = [oracle] ? LIMSJR sql your-name_dba sql> alter tablespace labware read only; exit [oracle]$
Export the database content
cd /backup/export ls -l time expdp your-name_dba directory=dmpdir full=y logfile=limsjr_full-###-export.log dumpfile=limsjr_full-###.dmpdp
Make the database read-write again
sql your-name_dba Password: sql> alter tablespace labware read write; exit [oracle]$
Test the export file
Carefully read the limsjr_full-###-export.log
in k1:/backup/export
.
- Are there any errors? No is good. If present, find cause, maybe re-rrun.
- Review the number of samples for LIMS.SAMPLE, LIMS.TEST, LIMS.RESULT, OPS.RIS_DATA.
Log into the SHIPTEST instance. Run these commands to confirm ability to restore content from the file generated.
ssh oracle@k1.ship.iodp.tamu.edu . oraenv ORACLE_SID = [oracle] ? SHIPTEST [oracle]$ time impdp your-name_dba directory=dmpdir logfile=exp###-test-export.log dumpfile=limsjr_full-###.dmpdp schemas=geodp### tables=lims.sample, lims.test remap_schema=lims:transfer schemas=geodp### remap_schema=geodp###:geodp###_review
Use SQL Developer to review the content of the SHIPTEST TRANSFER.NEW_SAMPLE table. Diffo for the restored copy of the GEODP### catalog.
Cleanup or leave it for your counterpart.
Notify everyone that its done
Post the export log file on Slack jrso-developer. Note the size of the export file. We will not zip it, nor send it over the wire.
Email it to
- MCS - jr_mcs@ship.iodp.tamu.edu
- IT_OPS - it_ops@iodp.tamu.edu
- Data Librarian - database@iodp.tamu.edu
- programmers - programmers@iodp.tamu.edu
- DBAs - sunil@rajaconsultancy.com, murli@rajaconsultancy.com
Note the size of the file and its location--if we're putting it in the right place, there's no need for the MCS to move it.
Take a copy of the export file
Login into the Oracle account at the ODA using WinSCP. Bring down a copy of the export file and its log to
- your laptop--or other transfer device; and
- stage a copy in \\novarupta\vol3\dml\oracle-data\
for future local reference (or until we have to clean/preen for space.
6. Provide courier services
The backup tapes created by the MCS return to HQ as soon as possible. If your travel itinerary supports that goal, let them know you can provide courier services.
1. Notify everyone. Database snapshot in progress.
- Notify jr_developer and the shipboard technical staff that the EOX database backup is in progress.
- The database is available for reports, but content cannot be changed while this activity is in progress.
2. Make the database read only.
Connect to the BUILD box. Execute all of the following steps from that system.
At the SQL>
prompt, run the last command.
Stop. Leave this window open for use below.
Technical note. This command prevents write access to any table content that resides in the storage facility under the "labware" table space. In our database implementation, this happens to be all tables owned by the LIMS schema. Use of Drill Report ("ops" schema) is not affected. Aspects of DescLogik software configuration are also still open to maintenance, though use of DescLogik itself is locked out by this change.
3. Full snapshot as oracle@k1.
Run another PuTTY session.
Connect to oracle@k1.ship.iodp.tamu.edu
From the prompt run these commands--change the expedition and yourname (will require logging in as a DBA User see PWSAFE) - - - JON Put Your Password in "QUOTES"
NOTE: While performing the EOX for expedition 384, it was discovered that the path represented by "data_pump_dir" did not exist. You can discover what the data_pump_dir is supposed to be using this query (logged in as your DBA account):
select * from dba_directories
;
The results have a lot of stuff that's not useful, but the DATA_PUMP_DIR should be there. Currently it is /backup/LIMSJR/dpdump. Note that "backup" is singular (it used to be plural) and there is apparently some desire to change this, so keep that in mind. If you find that the folder does not exist, the command in this step will fail and you may need to get an MCS or a DBA on shore to help create it.
You can also try:
create directory data_pump_dir as
'/backup/export'
;
grant read, write on directory data_pump_dir to transfer, system;
10/5/2021: Folder did not exist, as above, but MCS suggested I try creating it myself (in Putty, logged in as Oracle) using mkdir commands. That initially seemed to work, but it broke some kind of symbolic or virtual link that caused the file to be written to the wrong physical location (with reduced disk space), so not recommended.
|
Statistics. Typical durations for the last step from previous expeditions. 344S: 16 min 23 sec. 345: 13 min 54 sec. 346: 18 min 46 sec. 341S: 11 min 42 sec. 351: 12 min 9 sec; 360: 16 min 57 sec; 363: 3 min 21 sec; 367: 3 min 38 sec.
ODA for 3 expeditions of data (11.1 GiB): 4 min 17 sec.
396: 6 min 54 sec
4. Restore database read/write capability.
From the prompt [oracle@k1 ~]$
run these command, supply the appropriate credential
|
Stop. Leave this window open for use below.
5. Compress the exports for transport. Cleanup.
From the prompt [oracle@k1 ~]$
run these commands
|
Cut and paste the digest keys and file listing into a notification email. Enables remote users to verify the integrity of the files.
Please log these in a txt file in the \AD\support\'expedition named sub directory'\ see other expeditions for example.
Statistics. 367: 22 min to compress both files (~5 GiB to < 1 GiB); 345: 22 min 53 sec for bzip step. 351: full 137 min 46 for ~8 GiB content. 360: 67 min 27 sec for ~5 GiB file. 45-60 min for high recovery expeditions.
Exp 396: we had trouble initially because we ran out of disk space. This was apparently due to my creating the /backup/LIMSJR/dpdump folder under the Oracle account. This broke some kind of link-up that gives the backup folder lots of additional drive space. Don't do that.
Need to test DMPDP.
Need to post log results.
6. Notify everyone. Database snapshot process is complete.
Notify jr_developer and expedition technical staff that the process is done.
Speak with the MCS. They will pick up the data from here.
|
The MCS pick up the full export from oemjr:/backup/export
.
Inquire with the MCS. Ensure the (above) database content and the (below) ASMAN content are being taken to media for transport to HQ.
- Establish with MCS, LOs, CoChiefs when the final database snapshot will be taken for EOX.
- Circulate through the labs.
- Honor moratorium. Once content is uploaded to LIMS, and raw files have gone to data1, expedition-specific data may be cleaned off of instrument hosts and workstations. Good practice to confer with technical staff that manage this for their labs. There is variance between crews as to how these procedures are carried out.
- Conduct end of expedition procedures for backing up the database.
- Provide courier services if called upon to do so.
- Confirm with the MCS what is to be included in the backup going to shore and that it does cover all the information you are aware of that should go to shore.
- Ensure all your code changes are checked in.
- Clean the development office. Assist in the general cleaning efforts.
- Be ready to move out. Your replacement will be here soon.
- Provide assistance and information to oncoming developers.
The goal of all oncoming personnel is to ensure the function of laboratory and computing systems to support the successful execution of the new expedition.
Assist with those goals.
They may not have sailed for quite a while, plan to- Review your changes.
- Review the current state of the labs.
- Review the current state of deployed applications.