\PassOptionsToPackage{unicode=true}{hyperref} % options for packages loaded elsewhere
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provides euro and other symbols
\else % if luatex or xelatex
  \usepackage{unicode-math}
  \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage[]{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\IfFileExists{parskip.sty}{%
\usepackage{parskip}
}{% else
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
}
\usepackage{hyperref}
\hypersetup{
            pdftitle={CSci 658-01: Software Language Engineering Python 3 Reflexive Metaprogramming Chapter 1},
            pdfauthor={H. Conrad Cunningham},
            pdfborder={0 0 0},
            breaklinks=true}
\urlstyle{same}  % don't use monospace font for urls
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}
% Redefines (sub)paragraphs to behave more like sections
\ifx\paragraph\undefined\else
\let\oldparagraph\paragraph
\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
\fi
\ifx\subparagraph\undefined\else
\let\oldsubparagraph\subparagraph
\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
\fi

% set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother

\usepackage{caption}
\DeclareCaptionLabelFormat{nolabel}{}
\captionsetup{labelformat=nolabel}

\title{CSci 658-01: Software Language Engineering\\
Python 3 Reflexive Metaprogramming\\
Chapter 1}
\author{\textbf{H. Conrad Cunningham}}
\date{\textbf{29 April 2018}}

\begin{document}
\maketitle

{
\setcounter{tocdepth}{5}
\tableofcontents
}
Copyright (C) 2018, \href{http://www.cs.olemiss.edu/~hcc}{H. Conrad
Cunningham}\\
Professor of \href{https://www.cs.olemiss.edu}{Computer and Information
Science}\\
\href{http://www.olemiss.edu}{University of Mississippi}\\
211 Weir Hall\\
P.O. Box 1848\\
University, MS 38677\\
(662) 915-5358

\textbf{Advisory}: The HTML version of this document requires use of a
browser that supports the display of MathML. A good choice as of April
2018 is a recent version of Firefox from Mozilla.

\setcounter{section}{0}
\newpage

\hypertarget{introduction}{%
\section{Introduction}\label{introduction}}

\hypertarget{metaprogramming}{%
\subsection{Metaprogramming}\label{metaprogramming}}

Basically, \emph{metaprogramming} is writing code that writes code.

\begin{description}
\tightlist
\item[\textbf{Metaprogramming}:]
the writing of computer programs that can treat computer programs as
their data. A program can read, generate, analyze, and/or transform
other programs, or even modify itself while running (Adapted from
{[}Wikipedia 2018b, 2018c{]} and other sources)
\end{description}

We often do metaprogramming in our routine programming tasks but do not
call it that.

\begin{itemize}
\item
  Our web applications may generate HTML, JavaScript, and CSS code to
  enable development of a browser-based user interface.
\item
  Our Java programs may use \texttt{instanceof} to check the type of
  objects or otherwise manipulate itself with the Java reflection
  package.
\item
  Our C programs may use macros to define new features in terms of
  existing features.
\end{itemize}

Under the above definition, much of our study of domain-specific
languages uses metaprogramming.

\begin{itemize}
\item
  The \texttt{pic} little language processor takes a program expressed
  in an external textual language that describes a picture and generates
  output expressed in another language that gives instructions to a
  display program {[}Kernighan 1984{]}.
\item
  Several of the \href{../658lectureNotes.html\#StateMachineDSL}{State
  Machine DSL} processors read and parse a program written in a
  special-purpose textual language, represent the program internally in
  a semantic model, and then ``execute'' the model on inputs.
\item
  Other of the \href{../658lectureNotes.html\#StateMachineDSL}{State
  Machine DSL} processors use the semantic model to generate a
  ``program'' in another language such as C or the Graphviz dot language
  for graphs.
\item
  The \href{../658lectureNotes.html\#CompConfigDSL}{Computer
  Configuration} and
  \href{../658lectureNotes.html\#EmailMessageDSL}{Email Message}
  internal DSLs use the host language itself to encode special-purpose
  languages. The processors can then read and parse descriptions written
  in these special-purpose languages and manipulate the resulting data
  structures similarly to the external DSLs.
\item
  The Lua- and Python-hosted
  \href{../658lectureNotes.html\#LairDSL}{Lair Configuration} and the
  Ruby-hosted \href{../658lectureNotes.html\#SurveyDSL}{Survey} internal
  DSLs manipulate the structure of the processing programs themselves to
  implement the special-purpose language.
\end{itemize}

\hypertarget{reflexive-metaprogramming}{%
\subsection{Reflexive Metaprogramming}\label{reflexive-metaprogramming}}

The internal Survey DSL and Lair Configuration DSL are examples of
\emph{reflexive (or reflective) metaprogramming}.

\begin{description}
\tightlist
\item[\textbf{Reflexive metaprogramming}:]
the writing of computer programs that manipulate themselves as data.
(Adapted from {[}Wikipedia 2018b, 2018c{]} and other sources)
\end{description}

This manipulation may be at compile time, involving a phase of
transformations in the code before the final program is generated. Or it
may be at runtime, involving manipulation of the program's metamodel or
generation of new code that is dynamically executed within the program.

The \href{../658lectureNotes.html\#SurveyDSL}{Survey DSL} is a Ruby
internal DSL. It takes advantage of Ruby's metaprogramming facilities
such as the abilities to trap calls to undefined methods, add methods or
variables dynamically to existing objects at runtime, and execute
dynamically generated strings as Ruby code. It also uses Ruby's
first-class functions (closures) and flexible syntax -- although these
are not technically metaprogramming features of Ruby.

The \href{../658lectureNotes.html\#LairDSL}{Lair Configuration DSL}
programs use the metaprogramming features of Lua and Python in similar
ways.

Consider relatively common languages and their metaprogramming features.

\begin{itemize}
\item
  Java is a statically typed, compiled language. What are
  metaprogramming features available in Java?

  It has dynamic class loaders, a reflection API, annotation processing,
  dynamic method invocation (a JVM feature), JVM bytecode manipulation
  (mostly with external tools), etc. Java 8+ also has first-class
  functions and other features useful in metaprogramming.
\item
  Lua is a dynamically typed, interpreted language. What are the
  metaprogramming features available in Lua?

  It has metatables, metamethods, manipulation of environments, a debug
  library (introspection/reflection features), \texttt{loadfile} and
  \texttt{loadstring} functions to dynamically execute code, extensions
  in C, etc.
\end{itemize}

What about Python 3?

The reflexive metaprogramming features of Python 3.6 and beyond is the
primary topic of this set of lecture notes.

\hypertarget{why-study-reflexive-metaprogramming}{%
\subsection{Why Study Reflexive
Metaprogramming?}\label{why-study-reflexive-metaprogramming}}

In everyday application programming, we often use the products developed
by metaprogrammers, but we seldom use the techniques directly.

In everyday programming, use of reflexive metaprogramming techniques
should not be one of our first approaches to a problem. We first should
explore techniques supported by core language, its standard libraries,
and stable extension packages.

If no acceptable solution can be found, then we can consider solutions
that use reflexive metaprogramming techniques. We should approach
metaprogramming with great care because these techniques can make
programs difficult to understand and can introduce vulnerabilities into
our programs. We should design, implement, test, and document the
programs rigorously.

However, reflexive metaprogramming can be an important tool in a master
programmer's toolbox. If our jobs are to develop software frameworks,
libraries, APIs, or domain-specific languages, we can use these
techniques and features to develop powerful products that hide the
complexity from the application programmer.

Even when our jobs are primarily application programming, understanding
reflexive metaprogramming techniques can improve our abilities to use
software frameworks, libraries, and APIs effectively.

\hypertarget{reflexive-metaprogramming-in-python-3}{%
\subsection{Reflexive Metaprogramming in Python
3}\label{reflexive-metaprogramming-in-python-3}}

TODO: Update this to better reflect what the final notes cover and
include forward references as appropriate.

The reflexive metaprogramming features of Python 3 include:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Decorators
\item
  Metaclasses
\item
  Descriptors
\item
  Import hooks
\item
  Context managers
\item
  Annotations (e.g.~type hints)
\item
  Abstract Syntax Tree (AST) manipulation
\item
  Frame hacks
\item
  Execution of strings as Python 3 code (\texttt{exec}, \texttt{eval})
\item
  \emph{Monkeypatching} (i.e.~direct dynamic manipulation of attributes
  and methods at runtime)
\end{enumerate}

We have already used the final two in our implementation of
domain-specific languages. We will look at some of the others in these
notes. In particular, Chapter 3 looks at use of decorators and
metaclasses.

But, in the next chapter, let's first examine the basic features of
Python 3 upon which the reflexive metaprogramming features build.

\hypertarget{exercises}{%
\subsection{Exercises}\label{exercises}}

TODO: Decide if any are appropriate.

\hypertarget{acknowledgements}{%
\subsection{Acknowledgements}\label{acknowledgements}}

I developed these notes in Spring 2018 for use in CSci 658 Software
Language Engineering. The Spring 2018 version used Python 3.6.

Teaching a special topics course on ``Ruby and Software Development'' in
Fall 2006 kindled my interests in domain-specific languages and
metaprogramming. Building on these interests, I taught another special
topics course on ``Software Language Engineering'' in which I focused on
Martin Fowler's work on domain-specific languages {[}Fowler 2011{]}; I
subsequently formalized this as CSci 658. (I have collected some overall
ideas on \href{../DomainSpecificLanguages.html}{Domain-Specific
Languages} in a separate set of notes {[}Cunningham 2018c{]}.)

The overall set of notes on Python 3 Reflexive Metaprogramming is
inspired by David Beazley's \href{http://www.dabeaz.com/py3meta}{Python
3 Metaprogramming tutorial} from PyCon'2013 {[}Beazley 2013a{]}. In
particular, some chapters adapt Beazley's examples. Beazley's tutorial
draws on material from his and Brian K. Jones' book \emph{Python
Cookbook} {[}Beazley 2013b{]}.

Chapter 1 of the notes subsumes my previous notes on Metaprogramming
{[}Cunningham 2018b{]}.

I maintain these notes as text in Pandoc's dialect of Markdown using
embedded LaTeX markup for the mathematical formulas and then translate
the notes to HTML, PDF, and other forms as needed.

\hypertarget{references}{%
\subsection{References}\label{references}}

\begin{description}
\tightlist
\item[{[}Beazley 2013a{]}:]
David Beazley. \href{http://www.dabeaz.com/py3meta/}{Python 3
Metaprogramming (Tutorial)}, \emph{PyCon'2013}, 14 March 2013.
\item[{[}Beazley 2013b{]}:]
David Beazley and Brian K. Jones. \emph{Python Cookbook, 3rd Edition},
O'Reilly Media, May 2013.
\item[{[}Cunningham 2018b{]}:]
H. Conrad Cunningham. \href{../Metaprogramming.html}{Metaprogramming}
notes, revised 22 February 2018.
\item[{[}Cunningham 2018c{]}:]
H. Conrad Cunningham.
\href{../DomainSpecificLanguages.html}{Domain-Specific Languages} notes,
revised 2 April 2018.
\item[{[}Fowler 2011{]}:]
Martin Fowler. \emph{Domain-Specific Languages}, Addison Wesley, 2011.
\item[{[}Kernighan 1984{]}:]
Brian W. Kernighan. \href{http://doc.cat-v.org/unix/v8/picmemo.pdf}{PIC
-\/- A Graphics Language for Typesetting, Revised User Manual},
Computing Science Technical Report No.~116, Bell Laboratories, December
1984. {[}\href{../localcopy/picmemo.pdf}{local}{]}
\item[{[}Wikipedia 2018b{]}:]
Wikipedia,
\href{https://en.wikipedia.org/wiki/Metaprogramming}{Metaprogramming},
accessed 25 April 2018.
\item[{[}Wikipedia 2018c{]}:]
Wikipedia,
\href{https://en.wikipedia.org/wiki/Reflection_(computer_programming)}{Reflection},
accessed 25 April 2018.
\end{description}

\hypertarget{terms-and-concepts}{%
\subsection{Terms and Concepts}\label{terms-and-concepts}}

TODO: Update

Metaprogramming, reflexive metaprogramming.

\end{document}