CS229r:DualitybetweenBrascamp-LiebInequality&SubadditivityofEntropyJuspreetSinghSandhuandPrayaagVenkat1IntroductionTheBrascamp-Lieb(BL)inequalityisageneralinequalitywhichcapturesmanywell-knowninequalitiesasspecialcases.Theorem1(Brascamp-LiebInequality,generalform[BCCT08]).Let{Bi}mi=1beacollectionofsurjectivelinearmapsBi:RnRniand{ci}mi=1beacollectionofpositivescalarssuchthat:mXi=1cini=n.(1)Thenforanycollection{fi}mi=1ofnon-negativefunctionsfi:RniR0suchthatRxRni|fi(x)|dx<,thefollowinginequalityholdsZxRnmYi=1fi(Bix)cidxDmYi=1!ZxRnifi(x)dx"ci,(2)whereDisaconstantthatdependsonlyon{Bi}mi=1and{ci}mi=1.InTheorem1,{Bi}mi=1aretobetho* D E M O *jectionsofRnontolowerdimensional* D E M O *hly,thisinequalityallowsonetoupperboundtheintegralofaproductoffunctionsonlower-dimensionalspacesbytheproductofintegrals.Condition1isaconditionwhichensuresthatthe“dimensions”oneithersideoftheinequality“match”.InInequality2,theconstantDhasbeenexplictlycharacter* D E M O *lutionofacertainoptimizationprobleminvolving{Bi}mi=1and{ci}mi=1.Inthefollowing,wewillgiveseveralexamplestoprovideintuitionan* D E M O *thekeyaspectsofTheorem1.Hölder’sInequalityPickm=2,c1=1p1,c2=1p2forp1,p21andletu,v:RnR0befunctionssuchthatRxRnu(x)p1dx,RxRnv(x)p2dx<.Setf1=up1andf2=vp2a* D E M O *heprojectionsB1,B2tobetheidentitymap.Then,Condition1iseasilyseentobesatisfiedwhen1p1+1p2=1andInequality2simplifiestoHölder’sinequality:|hf,gi|≤kfkp1kgkp2.ThisexampleillustratestheflexibilityofCondition1;itallowsustochooseciinsuchawaysoastocontroltheleft-handsideofInequality2bydifferentnormsofthefunctions.Awell-knownspecial-caseofHolder’sinequalityistheCauchy-SchwarzInequality,whichcanberetreivedbysettingp1=p2=2.1
Loomis-WhitneyInequalityPickm=n,c1=...cn=1n1,andBi:RnRn1bethepro-jectionontotheithcoordinatehyperplane(meaningBi(x1,...,xn)=(x1,...,xi1,xi+1,...,xn)),fori=1,...,n.Again,Condition1isstraightforwardtocheck.Fori=1,...,n,takeanyfi:Rn1R0suchthatRxRn1fi(x)n1dx<.Then,Inequality2appliedtothefunctionsfn1isimplifiestotheLoomis-Whitneyinequality:ZxRnmYi=1fi(Bi(x))dxmYi=1kfikn1.Weremarkthatthisi* D E M O *usversionof)aspecialcaseofShearer’sLemma,see7.Shearer’sLemmaitselfcanberecoveredthroughdifferentchoicesof{Bi}and{ci}.ThisexampleillustratestwoimportantfeaturesofTheorem1.First,weprovestatementsaboutacollection{f1,...,fn}ofmanyf* D E M O *steadofjustm=2,asinHölder).Second,wecanchoosetheprojections{Bi}mi=1toreplaceanintegraloverRnwithintegralsoverRni,thelat* D E M O *aybeeasiertoevaluateinapplications.Moreover,the{Bi}and{ci}donothavetoallbethesame.TogetabetterunderstandingofwhattheLoomis-Whitneyinequalityissaying,letSRnbeameasurablesetandtakefi=I1/(n1)Bi(S)tobetheindicatorfunction(raisedtothe1/(n1)power)fortheprojectionofSontotheithcoordinateplane.NotethatifxSthenQni=1fi(Bi(x))=1.Thatis,ifapointisinthesetthenallofits(n1)-dimensi* D E M O *ionsareinthe(n1)-dimensionalprojectionsoftheset.Thismeans,thattheleft-handsideoftheLoomis-Whitneyinequalityisanupperboundonvol(S),wherevol(·)denotesthevolumeofasetinRn.Alongthesamevein,consideringeachofthetermsontheright-handside,wehavethatkfikn1=#ZxRn1IBi(S)$1/(n1)=(vol(Bi(S)))1/(n1).Substituting,theLoomis-Whitneyinequalitythenstatesthat:vol(S)nYi=1vol(Bi(S))1/(d1).Inotherwords,weobtainanupperboundonthevolumeofasetintermsofthevolumesofitslower-dimensionalprojections.Forexample,whenn=2,thissaysthattheareaofasetisatmostitslength(alongtheverticaly-axis)timesitswidth(alongthehorizontalx-axis).Theaboveinequalitygeneralizesthistohigherdimensions.ThisexampleillustrateshowTheorem1isoftenapplied;thefunctionsfiBiaretakentobeindicatorfunctionsoflowerdimensionalprojectionsaset,sothattheleft-handsideofInequality2representsthev* D E M O *etandtheright-handsideinvolvesvolumesofthelower-dimensionalprojections.SeethelecturenotesofBall[B+97]forfurtherapplicationsofthistechniqueinconvexgeometry.Application:CountingtrianglesinlargerandomgraphsWenowpresentanotherspecialcaseoftheBLinequalitythatwasrecentlyusedbyLubetzyandZhao[LZ15]toanalyzethenumberoftrianglesappearinginlargerandomgraphs.Specifically,oneofthetechnicallemmastheyprovedstatesthatiff:[0,1]2[0,1]isamea* D E M O *tionsuchthatf(x,y)=f(y,x),then:%%%%%Z(x,y,z)[0,1]3f(x,y)f(y,z)f(z,x)dxdydz%%%%%"Z(x,y)[0,1]2f(x,y)2dxdy!3/2.(3)2
Roughlyspeaking,thetripleofpairs(x,y),(y,z),(z,x)correspondstoatriangleinagraphandthequantityontheleft-handsideof3representsthedensityoftrianglesinacertainrandomgraphassociatedwiththefunctionf.Thisinequalityallowsonetoestimatethenumberoftrianglesinthisgraphbasedonthenumberofedges,whichisasimplerquantitytounderstandinthesettingof[LZ15].Tosolvetheproblemofcountinglargersubgraphsthantriangles,theyderiveageneralizedversionofin-equality3.However,itisstraightforwardtoseethatinequality3isjustaspecialcaseoftheLoomis-WhitneyinequalityandthegeneralizedversionisjustanextensionofHölder’sInequality,whichisagainaspecialcaseofTheorem1.1.1PastworkontheBrascamp-LiebinequalitiesInthissection,weprovideabriefsummaryofthehistoryoftheBLinequality.Originallyproveninvariouss* D E M O *byLieb[Lie02],Brascamp,LiebandLuttinger[BLL74]andBrascampandLieb[BL76],thegeneralformoftheBL-inequalitygiveninTheorem1isduet* D E M O *l.[BCCT08].Inequalitiesofthistypehaveapplicationinmathematicalphysics[CLL05],convexgeometry[B+97]andbeyond.WebrieflymentionaninterestingconnectionbetweentheBLinequalityandtheoreticalcomputerscience.Asdiscussedpreviously,theconstantDinInequality2wascharacterizedin[BCCT08]asthesolutionofanoptimizationproblem.Recently,Gargetal.[GGOW18]studiedthealgorithmictaskofcomputingD,giventheprojections{Bi}mi=1andconstants{ci}mi=1.AlthoughallspecialcasesofTheorem1consideredinthisreportsatisfytheinequalitywithD=1,untiltheworkofGargetal.,itwasnotknownhowtocomputeDingeneralinpolynomialtime.2SubadditivityofentropyWebeginbyintroducingthebasicsubadditivityofentropyinequalityandthenstatingaslightgeneralizationofit(Han’sInequality(5)),yieldingaconnectiontoShearer’sLemma(7).Weendwithageneralstatementaboutthesubadditivityofentropy(8).Theorem2(BasicSubadditivityofEntropy).ForrandomvariablesX1,..,Xn,weh* D E M O *wing:H(X1,..,Xn)nXi=1H(Xi)(4)Thisstatementisasimpleresultthatcanbederivedfromthechainruleofentropyandthefactthatcondi-t* D E M O *lydescreaseentropy.Equalityin4holdsiffXiXj,i,j[n].Thisstatementhintsataconnectionbetweenaglobalquantity(JointEntropy)andalocalquantity(projectedentropies).Weno* D E M O *Inequality(5),whichisaslightgeneralizationofthestatementaboveandisthekeyingredientinthespecialcaseofaversionofShearer’sLemma(7).Han´sInequalityHan’sinequalityallowsustocontrolthejointentropyofasystemviathemarginalentropies:H(X1,..,Xn)1n1nXi=1H(X1,..,Xi1,Xi+1,..,Xn)(5)Han’sinequalitycanbeprovedbyrewritingthejointentropyasaconditionalsumofentropies(forsomei[n]),droppingconditioningonallcoordinates>i,repeatingthisi,andthensummingupthetermsandapplyingthechain-ruleforentropy.3
Inthespiritofgeneralizingfurther,Han’sinequalitycanbeappliedtorelativeentropiesaswell.GivenafinitealphabetX,aproductdistributionP=nYi=1pionXn,andajointdistributionQonXn,itholdsthat:DKL(QkP)1n1nXi=1DKL(Q(i)kP(i))(6)whereP(i),Q(i)denotethemarginaldistributionsforP,Qrespectively.Shearer’sLemma&Han’sInequalityS* D E M O *maisastatementthatallowsustoboundthevol-umeofadiscretehigh-dimensionalobjectbylookingatitslow-dimensionalprojections.Theorem3(Shearer’sLemma).Givensomen,k,NZ+wherekn,anobjectF=(f1,..,fn)[N]n,andthek-dimensionalprojectionsFS={(fi1,..,fik)|ijS,S[n]},thefollowinginequalityholds:|F|(n1k1)YS[n],|S|=k|FS|(7)IfwereformulateShearer’sLemmabyreplacingthevolumeterms|·|in7bytheirentropiccounterparts,thenwerecoverHan’sInequality5whenk=n1,andBasicSubadditivityofEntropy4whenk=1.Thishintsataconnectionbetweensubadditivityofentropyandproblemsingeometrythatinvolveestimatinghigh-dimensionalvolumesbylookingatlow-dimensionalprojections.SincethestatementoftheBrascamp-Liebinequality(2)canbeinuitivelyinterpretedastryingtoboundthevolumeofahigh-dimensionalcontin-uousobjectviaitslow-dimensionalprojections,itisnaturaltoquestionwhethersubadditivityofentropy(initsgeneralform)isrelatedto2.Ageneralformofsubadditivityofentropy,whichsubsumes4,5and6is:Theorem4(SubadditivityofEntropy,[BLM13]).GivenindependentrandomvariablesX1,..,XntakingvaluesinacountablesetXwiththeproductdistribution:P=nYi=1piandZ=f:Xn[0,)apositive,boundedrandomvariable,thefollowinginequalityholds:Ent[Z]EP[nXi=1Ent(i)[Z]]1(8)Thisequationhasasimilarskeletalstructureto4,andbasicallyassertsthatthejointentropyofthesystemis* D E M O *abovebytheaverageofthesumofthemarginalentropies.ThegeneralizedEnt[Z]termallowsustocontrolhighermoments,formingthebasisoftheEntropyMethodforprovingconcentrationinequalities.21Ent[Z]isageneralizationofentropyandEnt(i)[Z]isamarginalofthegeneraliz* D E M O *neEnt[Z]=E[Zlog(Z)]E[Z]log(E[Z]),andthecorrespondingmarginal:Ent(i)[Z]=E(i)[Zlog(Z)]E(i)[Z]log(E(i)[Z]).Referto[BLM13]foracompletetreatment.2Toseehow8cantigthenaconcentratio* D E M O *,refertoChapter-4of[BLM13].Fortheex* D E M O *badditivitytomoregeneralentropies,refertoChapter-14in[BLM13].4
3DualityofsubadditivityofentropyandBrascamp-LiebInequalityWhilethereareseveralproofsofTheorem1,thefirstofwhichwasduetoBennettetal.[BCCT08],wenow* D E M O *ternativeapproachduetoCarlenandCordero-Erausquin[CCE09].Amongotherthings,theyshowthatInequality2isdual(inaprecisesensetobeexplainedbelow)toacontinuousversionof8(see9).ThisexactlyquantifiesandgeneralizestheconnectionwebrieflysawinSection2.3.1StatementofthemainresultWenowgiveoneofthemainresultsofCarlenandCordero-Erausquin,whichestablishesadualitybetweentheBLinequalityandsubadditivityofentropy.Foraprobabilitydensityfunctionf:RnR0withRxRnf(x)dx=1,wecandefinetheentropyasS(f)=ZxRnf(x)logf(x)dx.Infact,thisisactuallythenegativeentropy,butitwillbesimplertoworkwiththisversion.So,aswewillsee,"subadditivity"ofthisdefinitionofentropyreallymeanssuperadditivity.ForaprojectionBi:RnRni,weusefBitodenotethemarginalprobabilitydensityfunctionoffunder* D E M O *onBi3.Withthesedefinitions,wecanstatethemainresult.Theorem5(Theorem2.1of[CCE09]).Let{Bi}mi=1and{ci}mi=1beasinTheorem1.Thenthefollowingstatementsareequivalent:1.(BLinequality)Foranycollection{fi}mi=1ofnon-negativefunctionsfi:RniR0suchthatRxRni|fi(x)|dx<,Inequality2holds.2.(Subadditivityofentropy)Foranyprobabilitydensityfunctionf:RnR0withRxRnf(x)dx=1andS(f)<,thefollowinginequalityholds:mXi=1ciS(fBi)S(f)+logD.(9)HavingstatedTheorem5,wecanseethatbothInequalities2and9arestatementsthatrelateintegralsofafunctiontointegralsofitsmarginals.Moreover,notethatTheorem5preservestheconstantDwhenmovingbetween2and9.CarlenandCordero-Erausquin[CCE09]exploitthisfacttocharacterizethefunctionswhichmaketheBLinequalitytight.3.2DualrelationshipWecanconnectentropyandtheBLinequalitythroughthefollowingformulafrom[CCE09].Theorem6.LetfbeaprobabilitydensityfunctiononRnandφ:RnRbeanyfunctionsuchthatRxRneφ(x)dx<.Then,thefollowingholds:log!ZxRneφ(x)dx"≥hf,φi−S(f).(10)Thequantityo* D E M O *ndsideofInequality10isknowninothercontextsasthelogarithmofthemomentgeneratingfunction(orthecumulantgeneratingfunction)ofφ.Inequality10simplysaysthatthedual4ofentropyS(f)isthelogarithmofthemomentgeneratingfunction.3ThismeansthatiftherandomvariableXisdistributedaccordingtof,thenBiXisdistributedaccordingtofBi.4Formally,thisiscalledFenchel-Legendreduality,awell-studiednotiononitsownright.See[Roc15]forareference.5
Proof.Ourstartingpointistherelativeentropybetweenφandf,whichcanbesimplifiedtogettheright-handsideoftheinequalitywewanttoprove:ZxRnf(x)log"eφ(x)f(x)!dx=hf,φi−S(f),wherewehaveusedtheinnerproducthf,φi=RxRnf(x)g(x)dx.BytheconcavityofthelogarithmandJensen’sinequality,wecanupperboundtherelativeentropyas:ZxRnf(x)log"eφ(x)f(x)!dxlog"ZxRnf(x)eφ(x)f(x)dx!=log#ZxRneφ(x)dx$.Combiningthepreviousinequalitywiththeoriginalequalitygivestheresult.3.3ProofofTheorem5TheproofstrategyforTheorem5issimple.WewillassociatetheBLinequalitywiththeleft-handsideofFormula10andInequality9withtheright-handside,allowingustopassbackandforthbetweentwo.Proof.Forthefirstdirection,assumethatInequality2holdsandletfbeanyprobabilitydensityfunctiononRnwithS(f)<.Instantiatingthedualityformula10withφ=Pmi=1cilogfBiBiandrearranging,wehavethat:S(f)mXi=1ci#ZxRnifBi(x)logfBi(x)dx$log"ZxRnmYi=1fBi(Bix)ci!.Ontheright-handsideoftheaboveinequality,weapplytheBLinequalitytocontroltheintegralofproductssothatwegetZxRnmYi=1fBi(Bix)ciDmYi=1#ZxRnifBi(x)dx$ci=D,whereweusedthefactthateachfBiisadensityfunctionwhichintegratesto1.Substitutingthisintotheaboveinequality,wehave:S(f)mXi=1ci#ZxRnifBi(x)logfBi(x)dx$logD=mXi=1ciS(fBi)logD,completingthefirstdirectionoftheproof.Thereversedirectionissimilar,soweomitthedetails.Theproofillustratesthatthelogarithmofthemomentgeneratingfunctionmakesiteasiertoworkwithmarginals,sincetheexponentialofasumfactorsintoaproductofexponentials.4ConclusionInthisreport,wesawthattheBLinequalityisageneraltoolforboundingintegralsofproductsoffunctionsbyproductsofintegrals,whichariseinvariousapplications.ThemaintakeawayofTheorem5isthatwecannowproveBL-typeinequalitiesbymakinguseoftoolsfrominformationtheorytoprovethecorrespondingsubadditivityofentropyinequalities.Insomecases,thelattertaskissignificantlyeasier(asdemonstratedbyCarlenandCordero-Erausquin[CCE09]).Asasimpleexampleofthis,instantiateTheorem5with{Bi}ni=1astheprojectionsontothencoordinatehyperplanes.ThenInequality9becomesHan’sInequality5andInequality2becomestheLoomis-WhitneyInequality.6
AcknowledgementsTheauthorswouldliketothankProfessorMadhuSudanforhisfeedbackonthisreportandcorrespondingpresentation.References[B+97]KeithBalletal.Anelementaryintroductiontomodernconvexgeometry.Flavorsofgeometry,31:1–58,1997.[BCCT08]JonathanBennett,AnthonyCarbery,MichaelChrist,andTerenceTao.Thebrascamp–liebinequalities:finiteness,structureandextremals.GeometricandFunctionalAnalysis,17(5):1343–1415,2008.[BL76]HermJanBrascampandElliottHLieb.Bestconstantsinyoung’sinequality,itsconverse,anditsgeneralizationtomorethanthreefunctions.AdvancesinMathematics,20(2):151–173,1976.[BLL74]HermJBrascamp,ElliottHLieb,andJML* D E M O *eneralrearrangementinequalityformultipleintegrals.Journalo* D E M O *analysis,17(2):227–237,1974.[BLM13]StéphaneBoucheron,GáborLugosi,andPascalMassart.Concentrationinequalities:Anonasymptotictheoryofindependence.Oxforduniversitypress,2013.[CCE09]EricACarlenandDarioCordero-Erausquin.Subadditiv* D E M O *ropyanditsrelationtobrascamp–liebtypeinequalities.GeometricandFunctionalAnalysis,19(2):373–405,2009.[CLL05]EricCarlen,ElliottHLieb,andMichaelLoss.Aninequalityofhadamardtypeforpermanents.arXivpreprintmath/0508096,2005.[GGOW18]AnkitGarg,LeonidGurvits,RafaelOliveira,andAviWigderson.Algorithmicandoptimiza-tionaspectsofbrascamp-liebinequalities,viaoperatorscaling.GeometricandFunctionalAnalysis,28* D E M O *,2018.[Lie02]ElliottHLieb.Gaussiankernelshaveonlygau* D E M O *zers.InInequalities,pages595–624.Springer,2002.[LZ15]EyalLubetzkyandYufeiZhao.Onreplicasymmetryoflargedeviationsinrandomgraphs.RandomStructures&Algorithms,47(1):109–146,2015.[Roc15]RalphTyrellRockafellar.Convexanalysis.Princetonuniversitypress,2015.7