Z= L w/y (1-I x)/(1-I(1/y-1/x))

x=R/(L w)              i.e., R/(inductive reactance)
y=R/(1/(w C)) = RC w   i.e., R/(capacitive reactance)

calculate Z Z* = |Z|^2 and maximize by taking derivative wrt y (which is proportional to C)

a good way to calculate Z* is just replace I->-I

a=(1-I x)/(1-I(1/y-1/x)/y                                               
b=(1+I x)/(1+I(1/y-1/x))/y                                              

In[4]:= Simplify[a b]                                                           

                  2    4
                 x  + x
Out[4]= -------------------------
                  2    2       2
        -2 x y + y  + x  (1 + y )

In[5]:= Solve[D[%,y]==0,y]                                                      

                 x
Out[5]= {{y -> ------}}
                    2
               1 + x

%4 /. %5

              -2    2
Out[7]= {2 + x   + x }

So |Z| is =L w Sqrt[above] = L w (1/x+x)