MIKE Data Reduction [v1.6; December 2005]
MIKE Suggested Calibrations
- 10 Bias (0s) frames
- 10 trace flat images (per setup and side)
Note - It is ideal to
take a set of Arcs right before and/or after
creating the trace flat images!!
- 10 Milky Flats (per setup and side)
- Arc Calib:
- One pair taken at the same time as the trace flat
- One image taken before (or after) each science exposure
- One Standard star (each slit size)
Pre-Reduction Setup
mike_wrstrct ::
Write the MIKE structure to disk. In IDL you can modify the values
of any of the tags. You can then save the structure in fits form and
rewrite the ASCII file with the routine mike_wrstrct.
Example: IDL> mike_wrstrct, mike, FITS='mike_name.fits'
Time : fast
If you exit (or crash) IDL, you will need to read the structure back in.
mike_ar ::
Read the MIKE structure from disk.
If no name is given, the file looks for the first fits
file starting 'mike' that contains a '_'.
Example: IDL> mike = mike_ar()
Example: IDL> mike = mike_ar('mike_name.fits')
Time : fast
You know the code is working right when IDL> help, mike, /str
lists the structure.
Setup
- A unique setup is defined by the slit width, the binning, and the
angles of the echelle and prism. Obviously, each setup requires its own
set of calibration files (including Milky and Trace flats).
- The user must specify a setup number (>0) for each
file to be analyzed. If you have observed with only one setup
during the night, you may just do: IDL> mike.setup = 1
Or you can specify ranges in the mike structure:
IDL>
mike[20:30].setup = 2 (sets the 11 frames 0-indexed from 20-30 to setup 2)
- It is also important to make sure the objects are properly named
at this point. At the least, multiple exposures of the same object
should have identical object name (tag: mike.obj).
- mike_setup ::
This routine examines the mike structure and looks for calibration
files associated with the various setups.
It groups together exposures with identical Obj name (mike.obj) and
sets the obj_id tags accordingly.
A summary of the MIKE exposures is put in 'mike_summ.txt'.
-
Note, this program is useful to run if you happen to crash IDL
before saving the MIKE structure.
Example: IDL> mike_setup, mike
Time : Fast
- You know the code is working right when you
check the output file 'mike_summ.txt' and everything is
properly setup. Modify the setup and obj_id tags and rerun mike_setup
as necessary.
Calibrations
Create Bias (Zero) frames [Optional and NOT
recommended]
- The default method of bias subtraction is to use the overscan
region and the bias row. If the user prefers, one can also subtract
off a combined bias frame. This routine creates that combined frame
from a series of Bias frames. This routine
will create 1 bias frame per side per binning mode.
Alternatively, one may perform this step as a test of the bias
subtraction algorithm. The resulting images should have 0 counts
with no gross structure.
Example: IDL> mike_mkbias, mike
Time : 3-5min
- You know the code is working right when you
check the Bias frame : IDL> xatv, 'Bias/name.fits'
and it shows a generally blank image with no counts. Of course, there
is the possibility that you have bad bias frames and the code is still
working fine.
Set Gain
- We have found for the early years of MIKE (at least) that the
published gain values may not be appropriate for a given set of
observations. Therefore, we recommend resetting the gain according
to the value determined from a series of Milky flats.
- mike_setgain :: Loops on the Milky Flats and performs statistics (in mike_gain) to determine the gain and apply it to the gain TAG.
Example: IDL> mike_setgain, mike, setup, [side]
Time : 3-5min
- You know the code is working right when you see it looping
through all of the milky flat files and reporting gain values within
the ballpark of the published values. You might also print out the gain
tags after completion: IDL> print, mike.gain
Process Flats
- These routines process the Milky and Trace flats to create a
response image (pixel by pixel variations) and to determine the
curvature of the orders on the CCD. These files are essential to
run the MIKE pipeline. Note that the next three steps can be run
together using the routine
mike_allflat.
Before calling mike_allflat, however, you sould read the documentation
below and consider using several of the options.
- mike_mkmflat :: Creates a stacked, normalized milky flat
to correct pixel response variations.
The routine performs a running median normalization to each milky flat
and then stacks (averages) them together with rejection. This is a very
expensive routine. It is quite likely that one could use an archived
flat which nearly matches the setup in lieu of this step.
If one's milky flats have many absorption features in them (not recommended),
then one should call this routine with the /SMOOTH keyword.
Example: IDL> mike_mkmflat, mike, setup, [side], [/SMOOTH]
Time : 5min per flat image
- You know the code is working right when you see it looping
on the various milky flats. You can then check the flat with
IDL> xatv, 'Flats/Flat_B_01_M.fits'
- mike_mktflat :: Combines the
series of trace flats (currently
twilight flats) to create one high S/N image for order and slit tracing.
Because of the thermal gradients, it is highly recommended that one use
only a coeval set of trace flats.
Example: IDL> mike_mktflat, mike, setup, [side]
Time : 1min
- You know the code is working right when you see it process and
stack the various trace flat files. You should then
check the Trace Flat:
IDL> xatv, 'Flats/Flat_B_01_T.fits'
- mike_edgeflat :: Routine to trace the trace flat created above to determine the order curvature, and return a smooth fit
- The main driver is using
trace_crude on a 'sawtooth' image of the trace flat. At present
this is one of the weaker spots in the automated pipeline. This is
especially true given that there is no 'standard' prescription for
creating trace flats with MIKE. The main failing is that the code may
miss or misidentify the slit edges. As such, we highly recommend that you
run this routine with /INTER and (or) /CHK to interactively check the
order edges. It is particularly important that the code has the first
few orders correct before making an automatic attempt.
- We have recently (Mar 2005) turned the PCA fitting back on for the red
side. I suspect we turned it off for a reason initially, so this may
cause problems. You can turn off the PCA by using the /NOBASIS switch.
- The code also makes a first guess (usually incorrect) to the physical
order numbers for the orders.
- Example: IDL> mike_edgeflat, mike, setup, [side], /INTER, /CHK
Time : 1min
- You know the code is working right when the reduced_chi^2
for the fits are small (<0.005).
You can also check the QA plots in (e.g. 'QA/Flats01/qa_trcflt_01B.ps').
The X0 and PCA0 values should vary significantly but be well fit while
the PCA1-4 values should have minor variations.
- You can also check the tracing with
mike_chktrcflat. Along with
xatv, this routine is used to check the results from mike_trcflat.
IDL> xatv, 'Flats/Flat_B_01_T.fits'
IDL> mike_chktrcflat, mike, setup, 1, /NOSTOP, /FIT
- mike_allflat
:: The previous 4 routines have been bundled into
one simple script. We recommend using this routine once you are
comfortable that the code is working well with your setup, etc. We
recommend you
run mike_chktrcflat after the procedure is through to examine
the solutions.
Example: IDL> mike_allflat, mike, setup, [side, /INTER]
Time : <20min per side
- You know the code is working right when you go through all of the
checking steps listed above.
Arc Images
- General ::
The arcs are processed through a series of steps which:
- Process the raw frame (mike_procarc)
- Determine the offset between the trace flat and the position of the arcs due to thermal expansion (mike_arcalign)
- Derive a 1D solution down the center of each order (mike_fitarc)
- Create a 2D solution (mike_fit2darc)
- Traces the individual arc lines (mike_tracearc)
- Fits the changing slope of the arc lines (mike_fittrcarc)
- Create a 2D wavelength image (mike_mkaimg)
Because of shifts in the arc lines during the night (expected to
be due to thermal expansion), the arcs require special care. In
particular, one wants to associate each science frame with the
temporally closest arc image. This bookkeeping is done in mike_setup
and is repeated in the first step below.
-
mike_allarc: There are two modes of processing the arcs.
We strongly recommend against running the arc steps individually unless you
are debugging. In fact, it is possible that some procedures will not run
on their own at the present time.
- Process individually. This is generally only recommended
for redoing specific arcs. It is also useful for testing the code
on non-standard setups before running on all of the arcs.
The user inputs the indice(s) of the arc(s) to process.
Example: IDL> mike_allarc, mike, [indx], /INDX
Time : 5min per arc
- Process all together. This is recommended
If you are reducing a full night
of data, I suggest the latter.
Example: IDL> mike_allarc, mike, setup, [side]
Time : 5min per arc
- Procedures
- mike_setarcm:
For the processing to work, the code must first measure
the curvature of the arc lines. This involves processing one arc to near
completion using the code mike_setarcm. This procedure is called
automatically by mike_allarc and we recommend against calling
mike_setarcm directly. In any case, here is the calling sequence
(raw_fil is the name of an Arc file, e.g. 'Raw/mb0428.fits'):
Example: IDL> mike_setarcm, raw_fil, setup, side
Time : <10min
- You know the code is working right when
it runs to completion. Actually, if this code runs successfully then it
is very likely the rest of the arcs will process fine. This
code calls mike_procarc, mike_fitarc, mike_fit2darc, mike_trcarc and you should review
these codes (below) to see how to check for success.
- mike_procarc :: Process the Arcs.
This step bias subtracts (blue side only due to leakage in overscan
region on the red side) and flat fields the arc images. It chooses
the arc image closest in UT time to the science exposure. This routine
will also calculate the x,y shifts between the arcs being processed
and a 'template' arc. The `template' arc should be the arc taken
most closely in time to the trace flats. By default, the first Arc
of the night is taken as the template. Output
are fits files in the 'Arcs/' directory.
Example: IDL> mike_procarc, mike, setup, obj_id, [side]
Example: IDL>
rslt = mike_procarc_sngl('raw.fits', setup, side)
Time : 1min per image
- You know the code is working right when it finishes. It would
be quite unusual for this procedure to fail.
- mike_arcalign :: This routine
compares the Arc line profiles against the trace flat profiles to determine
an offset between the two. The offset is calcualte for a series of orders and
a 1st order polynomial fit is written to the arc_xyoff tag.
IDL> mike_arcalign, mike, setup, side, /CHK
- You know the code is working right when the output to the screen
shows a reasonable boxcar profile and the fractional shifts (indicated by the vertical
blue line) are smaller than 0.2 or so. Also, the total shifts should by on the
order of 2 unbinned pixels for the blue and less than 1 unbinned pixel for the
red.
- mike_fitarc :: This routine extracts a 1D spectrum down the
center of each order and determines the wavelength solution.
-
There are two levels of interaction with this routine. The most
interaction (not recommended) is to use /INTER which prompts the
user to identify and fit the Arc lines. One can also use the option
/PINTER which has the program attempt to identify a set of
lines in each order. The user than interactively fits the lines
using the routine x_identify which calls x1dfit.
As long as your arc lines are within several
pixels of my solution, things ought to run smoothly in the non-interactive
mode.
- At this point, we recommend using the full AUTO mode. If it fails,
we would hope to fix it.
The output is an IDL file containing the arc lines identified and
their pixel centroids: 'Arcs/Fits/Arc_name_fit.idl'.
The program can also create a ps file to examine the quality of fits.
Example: IDL> mike_fitarc, mike, setup, obj_id,
[side, /INTER, /PINTER, /PSOUT]
Example: IDL> result = mike_fitarc_work('arc_file',
setup, side)
Time : 5min per side per arc image
- You know the code is working right when the RMS of the fits to
each order are generally less than 0.1 pixel. There will be several exceptions
per full exposure. Also, it is common for the code to fail for the very bluest
orders of the blue side (at least with the original blue CCD).
The code produces a QA file (e.g. 'QA/Arcs01/qa_arcfit_mb0038.ps.gz')
which shows the residuals to the order by order fits including the RMS
(in pixels).
- mike_fit2darc :: This routine fits a 2D solution to the
wavelengths as a function of their y pixel value and their order number.
It is rather straightforward.
Example: IDL> mike_fit2darc, mike, setup, obj_id, [side]
Example: IDL> result = mike_fit2darc_work('arc_file',
setup, side)
Time : fast
- You know the code is working right when the RMS of the 2D fit
is less than 0.1 pix or so.
The code produces a QA file (e.g. 'QA/Arcs01/qa_arc2dfit_mb0038.ps.gz')
which shows the 2D solution and lists the RMS.
- mike_tracearc :: This routine traces arc lines in each
order. It traces the lines in the curved order frame of the original
image and then fits a straight line to each arc line. The slope
and centroid of the arc line is recorded to a file for later use.
Example: IDL> mike_tracearc, mike, setup, obj_id, [side]
Example: IDL> result = mike_tracearc_work('arc_file', setup,
side)
Time : 2min per side per arc image
- You know the code is working right by reviewing the QA file
it produces (e.g. 'QA/Arcs01/qa_tracearc_mb0034.ps.gz'). So long as the
arc lines and the fits look reasonably straight, everything should be fine.
- You can check the output by using
mike_chkarctrc
Example: IDL> xatv, arc_img
Example: IDL> mike_chkarctrc, mike, indx
- mike_fittrcarc :: Using the slopes derived in the previous step,
this routine fits for the slope throughout the 2D arc image. That is
the slope as a function of order and vertical height in the image. Uses
the usual least-squares algorithm.
Example: IDL> mike_fittrcarc, mike, setup,
obj_id, [side], /CLOBBER, /CHK
Example: IDL> result = mike_fittrcarc_work('arc_file',
setup, side)
Time : Fast
- You know the code is working right when RMS3 is less than 0.01.
The code produces a QA file (e.g. 'QA/Arcs01/qa_fittrcarc_mb0038.ps.gz').
- mike_mkaimg :: Using the 2D slope solution and the 1D arc
solution from step 2, this routine calculates a wavelength value for
every science pixel in the image.
The final wavelength image has air wavelengths and are saved as alog10
double values.
Output is : 'Arcs/Arc_nameI.fits'
Example: IDL> mike_mkaimg, mike, setup, obj_id, [side],
/CHK
Example: IDL> result = mike_mkaimg_work('arc_file',
setup, side, XOFF=)
Time : 1min
- You know the code is working right provided it does not
complain about an order not being single-valued in wavelength. Also
one can examine the BAD_ANLY parameter (it should be 0 upon completion).
Slit Profile
- mike_slitflat ::
This routine determines the slit profile
for each order. This is crucial for optimal sky subtraction, particularly
given the short slit length used in MIKE. The profile is used to correct
the illumination pattern of the science frames. The procedure is also important
for optimal extraction. It must be run after the arc calibrations as the
slope of the arc lines is a necessary input.
- A very important aspect of this routine is that the default mode is
to flatten the counts along the slit. That is, for an internal flat
we have found that there is a slope to the slit profile due to the non-uniform
illumination. Therefore, the default mode is to fit for this and divide
it out. If, however, you have observed the twilight sky, you will probably
want to turn the /NODETILT switch on.
Example: IDL> mike_slitflat, mike, setup, [side], [/chk], [/nodetilt]
Time : 10min per setup per side
- You know the code is working right when you check the QA
file (e.g. 'QA/Flats01/qa_slitflat_01B.ps.gz') and the slit profiles
look reasonably well behaved and the chi^2 values are around 0.5.
- If you want to see the fits in real-time, use /chk
Extraction
- The following routines all apply to a single object, i.e. multiple exposures of that object will be reduced together.
- Most of the following routines take the mike structure,
and the setup, side and obj_id tags.
- mike_allobj :: Run all extraction routines described below. Currently, it is preferable to run mike_allobj with these options:
IDL> mike_allobj , mike, 1, /procall, /nocr
Process the Image
- mike_proc :: Bias subtract and flat field the Raw image.
This routine takes the index number of the mike structure as input
or keywords setup and obj.
The index number is the integer in the first
column of the file 'mike.list'.
The resulting image is output in 'Final/' and is a
flattened flux and inverse variance fits file
(one gzipped fits image with two extensions per file).
Output is 'Final/f_name.fits'
Example: IDL> mike_proc, mike, SETUP=setup, OBJ=obj_id
Example: IDL> rslt = mike_proc_sngl('rawfil', [side])
Time : 1min per image
Check : xatv, 'Final/f_name.fits'
- You know the code is working right when you check the image
and it looks reasonable.
- mike_objcr (Optional, recommended with caution) ::
Compares two or more images to identify
cosmic rays. For two images, it uses the
difference of the two images scaled to the exposure times and
is quite conservative. One should NOT use this on
multiple exposures
of the same object if the object has moved along the slit or
if thermal expansion has resulted in a large shift in the data.
Example: IDL> mike_objcr, mike, setup, obj_id, side, [exp], [/NOCHK]
Time : 3min per pair
- You know the code is working right when you run with the
/CHK keyword and the code is not identifiying excessive CR's along the
object profile. This will happen (in particular) if the object has moved
significantly along the slit in between exposures.
Identify and Trace the Object
- mike_fntobj :: This code automatically identifies the object
in each order (rectify+collapse) and then traces using trace_crude.
A PCA analysis is then performed on the coefficients of each trace to
determine a smoothed (quasi-2D) solution that is useful for interpolation
and extrapolation. The code then creates and Object structure
mikeobstrct that contains the trace and will contain the 1D
extractions. Output: 'Extract/Obj_name.fits'
Example: IDL> mike_fntobj, mike, setup, obj_id, side,
/CHK
Time : fast
- This code creates an object structure in the Extract/ directory
(e.g. 'Extract/Obj_mb0033.fits.gz') which holds information relating
to the trace and extraction of the spectra.
Here is a
link to the code which lists the tags used.
- You know the code is working right when you run with the
/CHK keyword and the fit to the position in the slit fraction is nearly
constant and well fit as a function of trace number. Also, the image
with the trace overplotted should look sensible. Alternatively (or
in addition), check the QA output (e.g. 'QA/Obj01/qa_fntobj_mr0022.ps.gz').
The X0 and PCA0 coefficients should be well fit and the remaining PCA
coefficients should show small variation. Finally, the reduced_chi2 should
be < 0.1 (blue) or <0.01 (red).
Sky Subtraction
- mike_skysub :: Performs sky subtraction on an order by order
basis using a bspline fitting algorithm to all sky pixels in a given
order (the object is masked). The sky spectrum is also cross-correlated
with the UVES sky line list to investigate a shift. At the moment,
while it calculates the shift, it does nothing with the value.
- Output: Sky subtracted 2D image appended to the unsubtracted
flux and the variance 2D images (e.g. 'Final/f_name.fits'). Also
the bspline coefficients are written to 'Sky/sky_name.fits'
Example: IDL> mike_skysub, mike, setup, obj_id, side, [/CHK, /FCHK]
Time : 10min per exposure
- Examine the final product (and get wavelength info) by:
IDL> xatv, 'Final/f_name.fits', getsky=2,
WVIMG='Arcs/Arc_nameI.fits'
- You know the code is working right when you run with /FCHK
and the final image shows 'random' noise in places where sky lines
used to be. You can also run with /CHK to examine the order by order fits.
Finally, the QA file (e.g. 'QA/Obj01/qa_skysub_mr0022.ps.gz') shows
a zoom in on key sky lines in the red or several small patches in the
blue.
Extraction
- mike_box ::
This procedure first performs a boxcar extraction of all orders and
uses this result to gauge the SNR in each order. The routine than
loops on each order in decreasing SNR and fits a unique profile within
each order. This profile and a uniform sky is fit to the entire
order and the object flux is optimally extracted.
- Output: The tags .box_fx, .box_var, .box_wv, .fx,
.var, .wave are all filled
Example: IDL> mike_box, mike, setup, obj_id,
side, /RESCHK
Time : 15min per exposure (longer for high SNR data)
- Examine the final product:
IDL> mike_specplt, [/BOX]
- You know the code is working right when the reduced chi^2
values are 1.2 or lower (a bit higher for very bright objects).
Standard Star (Optional, and recommended)
- At this step it is recommended to process a standard star to
'test' the performance of extraction and to create a sensitivity
function used to flux the data (currently set to only use the calibration
files archived on the ESO web site). This is not required, however. The
user can used an archived sensitivity function or forgo fluxing
altogether. The next sequence of steps are roughly the same as what
will be performed on individual science objects. Greater detail
is given on these procedures below. Note that the variable index
below refers to the index number of the standard in the mike structure.
Also, you will need to use the /STD keyword for most of the routines.
- mike_proc :: Process the raw image (bias sub, flatten).
Example: IDL> mike_proc, mike, index
Time : 1min
- mike_fntobj :: Identify the object in the slit and trace it.
Example: IDL> mike_fntobj, mike, setup, index,
side, /STD
Time : 2min
- mike_skysub :: NOT RECOMMENDED. Sky subtract the image.
Example: IDL> mike_skysub, mike, setup, index, side,
/STD, /CHK
Time : 2min
- mike_box :: Extract the standard star (boxcar and optimal).
Example: IDL> mike_box, mike, setup, index, side,
/STD, /CHK, /SKIPSKYSUB
Time : 15min
- mike_calibstd :: Calibrate the extracted 1D standard star
spectrum. At present, this step requires a calibration file from the
ESO web site:
Grab the appropriate file (e.g. fhr4468.dat) and go. This will create
a sensitivity function in the Extract/ directory which can be used
to flux science objects.
Example: IDL> mike_calibstd, mike, index,
ESOFIL='fhr4468.dat'
Time : 3min
Flux Calibration and Coadding
Flux
- mike_flux :: This routine fluxes the data using a sensitivity
function. Use either the one you have created for that given night (set
using the FLUXFIL keyword) or an archived solution (if it exists for
the binning and orders of interest).
- Output: The tags .flux and .sig are filled in the Object
structure.
Example: IDL> mike_flux, mike, setup, obj_id,
side, FLUXFIL=, /STD
Time : 1min
- Examine the final product:
IDL> mike_specplt, /FLUX
Combine
- mike_combspec :: This routine
combines multiple exposures of the same object.
Should be run even on an object with a single exposure, to obtain 1d spectra.
- Output: A .fits file in the Spec/ directory
Example: IDL> mike_combspec, mike, setup, obj_id,
side, [exp_id]
Time : 1min
- Note: the documentation for this routine should be checked
2d -> 1d
- mike_1dspec :: This routine
combines orders into a single spectrum from the output of mike_combspec.
- Output: A .fits file in the Spec/ directory
Example: IDL> mike_1dspec, mike, setup, obj_id,
side, [exp_id], [LIST=]
Time : 1min
- Note: the documentation for this routine should be checked
Last modified 2005-12-18
Older code
mike_fittflat :: Performs a 2D fit to the traces created
in the previous step. This gives the full 2D optical model for each
side of the instrument. The relevant information is written
in the order structure in the Flats/ directory (e.g. OStr_B_01.fits).
Example: IDL> mike_fittflat, mike, setup, [side]
Time : 1min
- You know the code is working right when the RMS for the edges
are less than 0.1 (blue) and 0.01 (red). The code produces a QA file
(e.g. 'QA/Flats01/qa_fittflt_01B.ps') which also lists the RMS values.
- You can also check the fits as above but with the /FIT keyword
enabled.
IDL> xatv, 'Flats/Flat_B_01_T.fits'
IDL> mike_chktrcflat, mike, setup, 1, /NOSTOP, /FIT