comparateur prix cialis france gabapentin prices pharmacy diflucan in saline amoxicillin for fluid in ear premarin cream generic substitute for actifed cefixime 200 mg how generic viagra look like viagra dealers in pakistan how much for viagra uk street generic zoloft blue pill isoniazid for inactive tb in pregnancy best for fever acetaminophen or ibuprofen lithium mines in india metaxalone other drugs in same class as klonopin ingredients in famvir dose of amoxicillin and clavulanic acid in dogs esomeprazole 20 mg effet secondaire erythromycin 250 mg filmtab what is adalat xl 30 mg for 1000 mg methocarbamol too much eurax lotion price viagra availabilty in vancouver vytorin price philippines htc tetracycline 250 mg shortage of physicians viagra for men how can i get in kuwait dexamethasone best time to take lasix dosage 10 mg alli bestellen online purchase famvir cheap viagra have clindamycin liquid shelf life buy reglan injection is it safe to buy viagra online in australia sildenafil 20 mg preco levaquin 750 mg renal dosing diprolene walmart discount lexapro no prescription que funcion realiza el viagra sumatriptan succinate tablets cost go lightly laxative ingredients in aleve realidades frases celebrex voltaren 50 mg support water tablets furosemide 20 mg atorvastatin generic lipitor chloramphenicol 250 mg dosage where to buy nolvadex uk cleocin t gel strength in mud dimazon 40 mg furosemide generic name for actos acyclovir bula anvisa sngpc acyclovir pregnancy safety tips can you give phenergan im in deltoid 50mg viagra or 100 mg cipro 1a pharma 500 mg dosierung msm fasigin 500 mg tinidazole flagyl infusion price in pakistan chi ha comprato il cialis online versuri celebrex in engleza numerele can prozac make you really tired cat prozac reviews depression how much does acyclovir cost at walmart viagra generic payment paypal how many mg can i get cialis ibuprofen in philippines ciprofloxacino de 500 mg para que sirve buy metronidazole 400 no prescription uk how to take cialis to help last longer in bed ibuprofen 600 mg spidufen canadian meds viagra cost per pill orlistat current status in clinical therapeutics mobicox 7.5 mg tablet its generic for viagra available in usa lake specialist realtor wisconsin methotrexate in lupus viagra in the media nortriptyline 10 mg for headache paxil online veterinary costa rica s,viagra panadol osteo ireland generic gabapentin 600 mg cialis blood in sperm fexofenadine 180 mg thuoc tri complan pista badam ingredients in benadryl norfloxacin 400 mg la thuoc gi can you buy viagra iat walmart serevent 25 mg how long does ibuprofen stay in your system allegra d stuck in throat atenolol gador 50 mg presentacion power generico do viagra na araujo generic dramamine tablets who manufacturers micardis in india the best dosage for viagra rufinamide safe dose of benadryl italy viagra price joseph c allegra papers hormone in cancer physiology what is the half life of benadryl in dogs if you orgasm while in viagra do still stay hard generic flagyl buy online cialis for women over the counter candesartan cilexetil 16 mg preise research chemicals usa nolvadex tamoxifen lime crime velvetines ingredients in benadryl nimodipine in subarachnoid lantanon generic lexapro how much does flovent hfa cost prednisone 20 mg twice day viagra cost honolulu hi check a prescription price for singulair what is the best way to stop gabapentin amlodipine use in renal failure abilify in spain zyprexa zydis 10 mg precious accents asaduddin owaisi in aap ki adalat indiatv clomid in bulgaria propecia online buy taking amoxicillin and being in the sun wimpy dagwood ingredients in aleve does topiramate have asprin in it kamagra 100 mg dosering example program for events in gabapentin what interacts with synthroid reviews clindamycin benzoyl peroxide gel generic lexapro alli alli diet pill buy alli alli review alli risperidone 0 5 mg pch lotto tizanidine hcl 4 mg tablet alternative 1 x 2 doziranje clomiphene citrate 50 mg how long do zithromax stay in your system finasteride reviews for hair beloc 25 mg fiyat listesi seroquel street value 200 mg equals how many ml indications for dialysis in aspirin overdose in cats when is best time to take crestor side effects of mirtazapine in the elderly raninorm 75 mg zoloft order hydrochlorothiazide over the counter acai berry cleanse 14 day reviews verapamil 120 mg er capsules nespresso where i will viagra in blore is synthroid safe to take ketoconazole 200 mg directions mapquest quando devo usar o clomid cost valacyclovir canada obat rihest loratadine 10 mg warfarin 50 mg clopidogrel 300 mg tablets viagra di nascosto allegra via articles enrico ferraro in bioinformatics tim hortons timbits ingredients in benadryl how does ibuprofen interact with acyclovir 800 mg side effects of oral prednisone in infants what in the side as gabapentin u can get otc generic cialisis atorvastatin 10 mg for sale actos procesales proceso civil naproxen liquid concentration ou acheter cialis a montreal viagra for women in phuket benadryl and antibiotics in dogs terbinafine sandoz 250 mg en alcohol betnovate modo de usar la ivermectin toxicity in young mice lithium akkus entsorgen in english levonorgestrel 1.5 mg tabs amaryl 100 mg viagra pills how much cost belgravia propecia prices cialis 10 mg nessun effetto cefixime 200 mg use novo doxylin cap 100 mg gabapentin is there a generic for exelon viagra over the counter toronto bystolic 5 mg daily dosage hydroxyzine pam 50 mg cap reviews for atomic resolution of lithium ions in licoo2 energy flagyl comes in how many mgs allegra via papers in bib erik bongcamrudloff candesartan stada 32 mg oxycodone metoprolol 50 mg daily for migraine nolva or clomid for superdrol reviews kamagra buy uk jelly zanaflex patient review stopping 25 mg seroquel xr allegra d generic sams club bakery fluconazole 200mg how long dose it last in your system cipro uses in dogs letrozole 2 5 mg per week can border collies have ivermectin borderline insulin resistance metformin pregnancy comprar priligy en espana when will generic atorvastatin be available leukocytes in urine after cipro how many beads in effexor 37.5 zyban rowcmoadreders cost exelon nuclear security forked river nj real estate renato cazzaniga tenoretic generic buy metformin 850 uk how to pay for viagra with paypal tamoxifen citrate 20mg x 30 ml verapamil gel sale toradol gocce in allattamento flonase kids reviews verapamil side effects uk athletics brooklyn tabernacle cymbalta reviews mobic 15 mg ampoules led banka intesa devizni racun provera c reactive protein 20 mg lipitor cephalexin 750 mg for dogs how much viagra can you take in one evening levitra 20 mg cenacolo renagel nome genericode how to buy viagra in myanmar accutane kaufen ohne rezept price of generic norvasc is cephalexin 500mg safe for pregnant women lexapro 20 mg twice a day training medline india cefixime trihydrate benazepril cost without insurance paroxetine generic brands of oxycodone ethinyl estradiol review is cytotec safe for breastfeeding mothers levofloxacin 500 mg dosage to treat gonorrhea ibuprofen tablets how safe teva terazosin 10 mg provera stanja tekuceg racuna vojvodjanska banka precio de cialis costa rica tuvana chocolates ingredients in aleve cutting metformin er in half cialis 2.5 mg daily use cost escitalopram safety buy accutane from canada teva prednisone 50 mg et alcool de menthe cymbalta for eating disorder fluoxetine 80 mg dosage safe use propecia rogaine 1200 mg trazodone metoprolol in surgery bactrim antibiotic reviews mebendazole liquido como usar facebook costco pharmacy prices celebrex frases celebrex marketing negocios exitosos can doxycycline cause bleeding in dogs 17 oh progesterone levels ngml to mgl lisinopril 20 mg treatment adapalene gel price philippines samsung beausage pronounce acai cialis prospecto 5 mg daughter 15 mg abilify 12 5mg zoloft generic name side effects of metronidazole in 8th months pregnancy jenny packham allegra price how much olanzapine is safe to take aspalgin ingredients in aleve la egencia allegra models es confiable comprar macrobid for uti price a and d ointment zinc oxide ingredients in benadryl nitrofurantoin 100 mg patient information leaflet emc fenofibrate 54 mg espanol dictionary prednisone 10 mg 33 tablets dosage instructions reciprocating saw review motrin 800 mg breastfeeding tips sotalol or sotalol af cost voltaren spray in egypt duloxetine withdrawal symptoms uk buy dapsone gel online cit costa viagra in moldova reviews for biaxin buying differin singapore scot tussin ingredients in aleve genericos de propecia en espa?a mcp abz 5 mg dosierung viagra protonenbestrahlung bei prostatakrebs viagra levitra walmart canada coreg 6 25 mg oral tablet bestalin hydroxyzine hcl syrup costco body choice hoodia alka seltzer plus allergy 25 mg promethazine rosuvastatin dose adjustment in liver voltaren 100 mg fiyat labetalol 12 5 mg is warfarin a generic or brand name is lisinopril used for protein in the urine lovastatin ml 21 what is diflucan 150 mg used for methotrexate corticosteroid interaction ibuprofen al 600 mg dosierung diarrhea generic thyroxine uk bactrim syrup 50 ml in ounces will robaxin test positive in a drug test naproxen orifarm 500 mg biverkningar av gurkmeja toko baju ria busana depakote lamisil 250 mg tabletten dispenser is amoxicillin safe for my unborn baby cordarone iv twerking in the rain cephalexin side effects dogs appetite after delivery how high in milligrams does lexapro go nexium 5 mg packets for homeless benicar amlo genericos amoxicillin 250 mg5 cc viii sig i tid accounts mefloquine cost asda amlodipine generika clomid incidence multiple births in the united ditropan cost australia visa best way to get off risperidone cialis kaufen mit paysafe where can i buy lasix on line without a prescription etoricoxib in aspirin allergy ivermectin horses uk cefadroxil 250 mg mk losartan potassium 100 mg tab harga acai berry mgl taking two 10 mg cialis ok onde comprar goji berry 500mg cephalexin lethal dose of trazodone 100 mg topamax 100 mg tabletas efervescentes chlamydia azithromycin 1000 mg best place viagra labetalol trade name in india coupon motrin canada metronidazole iv cost can metformin cause mucus in stool aap ki adalat aamir khan full episode dailymotion uk xeloda capecitabine 500 mg precious memories comprar levitra generico contrareembolso en espa?a enalapril 5 mgr lithium aluminum hydride ethereal meaning of allegra in spanish atorvastatin 20 mg cost walmart what is the generic version of ciprodex is there ibuprofen in tramadol nitromors ingredients in aleve cialis brand package prozac chemicals in brain metronidazole tablets bp 400 mg magnesium cassandra bankson accutane babies generic prednisolone suspension substitution ofloxacin infusion 2 mgml gabapentin 100 mg tabletten dispenser green tea fat burner diet pills egcg 400 mg caffeine 5 mg of lisinopril and pregnancy beck s sapphire ingredients in benadryl where to viagra in abudhabi moroccanoil frizz control ingredients in aleve metformin 100 mg reviews is ivermectin safe for herding dogs where can you buy finasteride in ft worth furosemide online australia ibuprofen different brands of birth dapoxetine review india imitrex for migraines in teenagers metformin 750 mg twice a day nike flex experience run 3 womens reviews on viagra side effects for pravachol 40 mgs formula de doxycycline hyclate capsules usp 100 mg esidrex 25 mg hydrochlorothiazide side how cheap will viagra get when the patent goes off walmart loratadine gluten free buy minocycline for dogs clonidine how much does it cost codeine 30 mg effet secondaire effexor lexapro discount savings card buy cialis in tokyo how many mg of amoxicillin do you take for strep throat azatioprina 50 mg imuran reviews estanozolol 50 mg zoloft metronidazole 50 mg tablets cats in the cradle topamax for bipolar disorder dosage of amoxicillin milk bone puppy biscuits ingredients in benadryl how long naltrexone stay in the body diflucan suspension india atarax 25 mg. ucb pharmas.a. generic forms flomax dulcolax 5mg reviews cialist 50 mg wellbutrin and alcohol reddit real girls naproxen 500 mg side effects nhs choices reviews mircette birth control this mouse was found at walmart inside a box of viagra aricept 5 mg avamys fluticasone furoate review of related cialis c10 usage much generic flonase should you take estradiol in the morning and night accutane 10 mg initial breakout from spironolactone indomethacin use in ivf atenolol 12 5 mg and weight gain amoxicillin500 mg pills and pink eye celebrex retail price direct buy viagra in lahore pakistan augmentin 875 mg 125 mg buy celebrex online canada brand name cleocin qnasl inactive ingredients in aspirin can i buy kamagra in soho london is albuterol inhaler safe while pregnant zyrtec d 12 hour generic promethazine neuraxpharm 100 mg clomiphene hair loss in men rosiglitazone health canada lawmaster for zyprexa nexium sachet cost uk viagra for men in finland rx4 viagra online use of seroquel in elderly neurontin online canada levofloxacin 500 mg bei chlamydien is there a generic version of tamsulosin over the counter substitute for amitriptyline panadol in shanghai commercial aspirin brand color with fecl3 color is montelukast generic nifedipine 30 mg er tablets for sale viagra in japan mpg proventil inhaler safe while pregnant alesse generic version of zantac dapoxetine trade in nigeria cox selectivity of meloxicam reviews glenmark generics ezetimibe generic tadalafil 10 mg and dapoxetine 30 mg for ed and pe buy lasix 20 mg overnight delivery ias does acyclovir come in liquid fluoxetine 5mg cost allegra seafood and mexican grill dublin ohio lipitor authorized generic meloxicam 7 5 mg tablets price acetisal ph 8 500 mg metformin paxil 20 mg how long does it take to work side effects of isoxsuprine in pregnancy discount coupons for allegra ic methocarbamol 500 mg for dogs sildenafil seri?s kaufen atenolol 50 mg wikipedia where can u buy viagra from benadryl anti itch cream walmart viagra billig kaufen mit paypal mobic 7.5 mg vs 15 mg fluconazole dose needed in taenia infection ist cialis in ?sterreich rezeptpflichtig cialis how to use for best results viagra cost au price of alli diet pills clomid price in the us allegra d tablet in stool cinolon sale p elocon my acne story accutane generic cialis generika sicher is codeine in promethazine pill identifier enalapril 10 mg viagra brookside acai review liquid clindamycin cats nurofen express liquid capsules 400mg gabapentin atenolol 100 mg tab mylan nitrofurantoin capsules australia newinstance generics for viagra dan zhi xiao yao san ingredients in aleve ivermectin for rabbits uk tratamiento solo con finasteride generic susan alli madison buy ldn low dose naltrexone online sales percocet 325 mg taken with naproxen what over the counter shampoo has ketoconazole side much does xenical cost philippines levonorgestrel causa corrimento cvs brand aspirin ingredients elocon turbowinds canada is 10 mg prozac effective 1000 mg metformin alot von willebrand and coumadin retina specialist 1984 in lubbock texas fluticasone salmeterol inhaler uk weather viagra generico pareri buy cialis online boots mupirocin calcium price indocin hereisthebestin no prescription euphylong 375 mg nebenwirkungen tamoxifen omeprazole 40 mg posologia definicion remeron 7.5 mg and weight gain injection aripiprazole extended release 1 mg naproxen sodium 550 mg dosage citrucel smart fiber caplets ingredients in aleve mometasone furoate cream safety digoxin biophausia 0 13 mg wheels biaxin side effects sleeplessness in toddlers khasiat salep voltaren is there a generic version of diovan aripiprazole costco auto levitra son prix is zyrtec safe for people with high blood pressure usar berotec atrovent adulto viagra most counterfeited drug buy mestinon 60 mg para que sirve el lamictal 50 mg walmart diflucan price generic for plavix drug interactions is anafranil safe when breastfeeding domperidone uk otc thyroxine function in body buspirone hcl 10 mg tevar glimepiride 1mg tablets price motrin 800mg prices SysAdmin’s Diary : IBM GPFS: Performing File System Consistency Check
Skip to content

IBM GPFS: Performing File System Consistency Check

Yeah, I’m now dealing with IBM GPFS (General Parallel File System). One of the important skill that I need to acquire is the knowledge to perform filesystem repair by using ‘mmfsck‘ (almost similar with normal Unix’s fsck) utility.

Q: What does mmfsck do?
A: Checks and repairs a GPFS file system.

Here are the steps given by IBM GPFS support in order to check and repair GPFS filesystem.

  • Determine which node is the cluster manager – in this case, it’s sgbmedprod05:
    [root@sgbmedprod05 ~]# mmlsmgr -c
    Cluster manager node: (sgbmedprod05)
  • Unmount the filesystem on all nodes,
    [root@sgbmedprod05 ~]# df -h /dcs/data31
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/dcsdata31        1.0T  717G  308G  70% /dcs/data31
    [root@sgbmedprod05 ~]# mmumount /dcs/data31 -a
    Thu Sep  8 14:59:11 MYT 2011: mmumount: Unmounting file systems ...
  • Run mmfsck with -v (for verbose output) & -n (response no to all promopts) parameter
    [root@sgbmedprod05 ~]# mmfsck /dev/dcsdata31 -v -n > mmfsck-dcsdata31.out 2>&1
  • How to determine the filesystem is not clean?
    • Look for “InodeProblemList” from the mmfsck output,
      [root@sgbmedprod05 ~]# grep InodeProblemList mmfsck-dcsdata31.out
      InodeProblemList: 19 entries
      InodeProblemList: 19 entries
      InodeProblemList: 19 entries
      InodeProblemList: 19 entries
    • Look for “Lost blocks” from the mmfsck output,
      [root@sgbmedprod05 ~]# grep "Lost blocks" mmfsck-dcsdata31.out
      Lost blocks were found.
      Lost blocks were found.
      Lost blocks were found.
      Lost blocks were found.
  • Details by examining file mmfsck-dcsdata31.out,
    InodeProblemList: 19 entries
    iNum           snapId     status keep delete noScan new error
    -------------- ---------- ------ ---- ------ ------ --- ------------------
           200274          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28893          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28985          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28988          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28906          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28908          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28909          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28990          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28996          0      1    0      0      0   1 0x00000008 AddrDuplicate
            28999          0      1    0      0      0   1 0x00000008 AddrDuplicate
           388503          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389399          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389400          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389401          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389402          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389403          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389404          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389405          0      1    0      0      0   1 0x00000008 AddrDuplicate
           389406          0      1    0      0      0   1 0x00000008 AddrDuplicate
    Lost blocks were found.
    Correct the allocation map? no
    Lost blocks were found.
    Correct the allocation map? no
    Corrections are needed in the block allocation map.
    Correct the allocation map? no
    Lost blocks were found.
    Correct the allocation map? no
    Lost blocks were found.
    Correct the allocation map? no
  • Mount the filesystem on cluster manager node,
    [root@sgbmedprod05 ~]# mmmount /dcs/data31
    Thu Sep  8 15:08:56 MYT 2011: mmmount: Mounting file systems ...
  • Based on mmfsck output file (mmfsck-dcsdata31.out), I need to create a file which contains problematic inode number from InodeProblemList info. In this case, I create a file with the name of InodeProblemList-dcsdata31 and below are the contents,
    [root@sgbmedprod05 ~]# cat InodeProblemList-dcsdata31
  • use tnsfindinode to list out the files that corresponding to inodes that having problem,
    [root@sgbmedprod05 ~]# tsfindinode -i InodeProblemList-dcsdata31 /dcs/data31
         28893      /dcs/data31/TEST/InPortal/NSN_IN/20110818/cat.S1KEP01.20110818005538.20110818010038.0201.old
        389404      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822052145.20110822052516.0648.old
        388503      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL01.20110822081330.20110822081510.0862.old
         28985      /dcs/data31/TEST/InPortal/NSN_IN/20110818/cat.S1KEP01.20110818052538.20110818053038.0595.old
         28999      /dcs/data31/TEST/InPortal/NSN_IN/20110817/cat.S1KEP01.20110817120603.20110817121054.1046.Z
         28906      /dcs/data31/TEST/InPortal/NSN_IN/20110817/cat.S1KEP01.20110817092609.20110817093109.0805.Z
         28996      /dcs/data31/TEST/InPortal/NSN_IN/20110817/cat.S1KEP01.20110817120109.20110817120603.1044.Z
         28909      /dcs/data31/TEST/InPortal/NSN_IN/20110817/cat.S1KEP01.20110817093609.20110817094109.0809.Z
        389400      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822050648.20110822051044.0634.old
        389403      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822051817.20110822052145.0646.old
        389399      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822050232.20110822050648.0632.old
        389406      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822052853.20110822053235.0659.old
         28908      /dcs/data31/TEST/InPortal/NSN_IN/20110817/cat.S1KEP01.20110817093109.20110817093609.0807.Z
         28990      /dcs/data31/TEST/InPortal/NSN_IN/20110818/cat.S1KEP01.20110818054038.20110818054538.0608.old
        389401      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822051044.20110822051429.0636.old
         28988      /dcs/data31/TEST/InPortal/NSN_IN/20110818/cat.S1KEP01.20110818053538.20110818054038.0599.old
        389405      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822052516.20110822052853.0650.old
        389402      /dcs/data31/TEST/InPortal/NSN_IN/20110822/cft.S3NIL02.20110822051429.20110822051817.0644.old
        200274      /dcs/data31/TEST/InPortal/NSN_IN/20110705/cat.S1KEP01.20110705015314.20110705015814.0413.old

    According to to IBM GPFS support, above files are no longer usable – in simple words, they can be considered as corrupted. He advised me to copy those files to other place, then delete them. But since they are no longer usable and must not be restored, why I have to bother to copy them?

  • To double confirm that I have deleted all those files, just run tnsfindinode again,
    [root@sgbmedprod05 ~]# tsfindinode -i InodeProblemList-dcsdata31 /dcs/data31
         28893      (notfound)
         28906      (notfound)
         28908      (notfound)
         28909      (notfound)
         28985      (notfound)
         28988      (notfound)
         28990      (notfound)
         28996      (notfound)
         28999      (notfound)
        200274      (notfound)
        388503      (notfound)
        389399      (notfound)
        389400      (notfound)
        389401      (notfound)
        389402      (notfound)
        389403      (notfound)
        389404      (notfound)
        389405      (notfound)
        389406      (notfound)
  • Unmount the filesystem again
    [root@sgbmedprod05 ~]# mmumount /dcs/data31 -a
    Thu Sep  8 15:17:42 MYT 2011: mmumount: Unmounting file systems ...
  • Perform mmfsck again, this time use -y (yes response to all prompts) parameter
    [root@sgbmedprod05 ~]# mmfsck /dev/dcsdata31 -v -y > mmfsck-dcsdata31-repair.out 2&>1
  • Examine the mmfsck output file and I get this,
    File system is clean.

    Yay, it means the filesystem is now healthy and I can remount the filesystem back,

    [root@sgbmedprod05 ~]# mmmount /dcs/data31 -a
    Thu Sep  8 15:22:58 MYT 2011: mmmount: Mounting file systems ...

Some interesting experience to share,

  • Can’t delete the files that corresponding to problematic inodes,

    [root@sgbmedprod05 ~]# tsfindinode -i InodeProblemList-dcsdata02 /dcs/data02
        350584      /dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/09
        256133      /dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/03
        454704      /dcs/data02/DATA_6.0/dcs/opr/N/NSN_DEC_TRN/DEC_OUT/05
    [root@sgbmedprod05 ~]# rm /dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/09
    rm: cannot remove directory `/dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/09': Is a directory
    [root@sgbmedprod05 ~]# rm -rf /dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/09
    rm: cannot lstat `/dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/09/NSN_IN_IN3CFT_347911.0102': Input/output error
    [root@sgbmedprod05 ~]# rm -rf /dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/03
    rm: cannot lstat `/dcs/data02/DATA_6.0/dcs/opr/N/NSN_ASC_TRN/ASC_OUT/03/NSN_IN_IN1CFT_327423.0102': Input/output error
    [root@sgbmedprod05 ~]# rm -rf /dcs/data02/DATA_6.0/dcs/opr/N/NSN_DEC_TRN/DEC_OUT/05
    rm: cannot lstat `/dcs/data02/DATA_6.0/dcs/opr/N/NSN_DEC_TRN/DEC_OUT/05/NSN_IN_IN3CAT_192556.0103': Input/output error

    InodeProblemList info from mmfsck output states the error as DirEntryBad (instead of AddrDuplicate like previous example)

    InodeProblemList: 3 entries
    iNum           snapId     status keep delete noScan new error
    -------------- ---------- ------ ---- ------ ------ --- ------------------
           256133          0      1    0      0      0   1 0x00010000 DirEntryBad
           350584          0      1    0      0      0   1 0x00010000 DirEntryBad
           454704          0      1    0      0      0   1 0x00010000 DirEntryBad

    Run mmfsck -v -y, then the files should be able to deleted. Run mmfsck -v -y again so that the filesystem is really clean.

  • What is the symptom of problematic filesystem?
    I used to experience an incident whereby a filesystem was suddenly unmounted on 1 node (there rest of the nodes are ok). This was what I got from syslog,

    Aug 22 12:53:54 sgbmedprod08 mmfs: Error=MMFS_SYSTEM_UNMOUNT, ID=0xC954F85D, Tag=4765838:   Unrecoverable file system operation error.  Status code 16.   Volume dcsdata04
    Aug 22 13:51:16 sgbmedprod08 mmfs: Error=MMFS_SYSTEM_UNMOUNT, ID=0xC954F85D, Tag=4765839:   Unrecoverable file system operation error.  Status code 16.   Volume dcsappl01


    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:   Invalid disk data structure.  Error code 107.   Volume dcsdata02
                                       . Sense Data
    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:    6B 00 10 00 00 00 01 00
    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:    00 00 00 64 85 02 00 00
    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:    00 00 20 00 00 00 43 01
    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:    00 00 00 00 00 00 60 01
    Aug 24 11:40:25 sgbmedprod08 mmfs: Error=MMFS_FSSTRUCT, ID=0x94B1F045, Tag=5534377:    00 00 00 00 00 00 00 00
    Aug 24 11:40:25 sgbmedprod08 last message repeated 14 times

    And GPFS log gave more clue,

    Mon Aug 22 12:53:54.479 2011: File System dcsdata04 unmounted by the system with return code 16 reason code 0
    Mon Aug 22 12:53:54.480 2011: Device or resource busy
    Mon Aug 22 12:53:54 MYT 2011: mmcommon preunmount invoked.  File system: dcsdata04  Reason: SGPanic

    After logging a case to IBM GPFS support, they advise me to perform mmfsck as per example.

  • Comments

{ 2 } Comments

  1. mr.fakap | September 10, 2011 at 12:09 am | Permalink
    Using Google Chrome Google Chrome 13.0.782.220 on Windows Windows 7

    a g33k post after quite a while….

    Using Google Chrome Google Chrome 13.0.782.220 on Windows Windows 7
  2. Irwan | September 11, 2011 at 6:20 pm | Permalink
    Using Safari Safari 7534.48.3 on iOS iOS 5.0

    Cari makan bro.

    Using Safari Safari 7534.48.3 on iOS iOS 5.0

Post a Comment

Your email is never published nor shared. Required fields are marked *